















2 ^ 

^V°-‘ oS- *"”"CV"y.>-‘'-<*' 

a t " 


‘‘‘j^ A O/'o • )i-‘'‘Jif^ * 

'^.V_ /^s5L" 


' O 
'*> 


. 

^ ^ <3 csi <« _ 


r '<K, >* 

^ £) N O ^ jk. ^ 

* ‘ ' 



o , .C> 


«r n ^ 




' 'V -^ *• 

* ^ov * 

; ^°^<l^, * 

^<11^ *^OnO'^ 

<J5r ^ 5 * 0> <x 1 

- y^ ^ 




«7 J\\ 83 /h, ^ ’y^ 

• - ° 

Lia ong^ ^ ^ ^ ^ ^ ^ ^ 

'^., \„/’ ‘"jfc*-V„'i‘' - 

V , v ^’■«ft % <5tt j^.‘> lLl^^ >~. ^ 

Xw/ -.Ilf / •<!-''X. X^.- 

o#i.-» J> „„„ "'o »>■'* ^ 

.^o » ' utn^ I ao '#w v j r^’X'^x tk^ ** \_^ - •■ 

« 



v<b 

to "j^ n 

■'X'wr-'v- < 


V r-) ik*. j* >>vSit^ a’ 

* % G,' 

O g" ®«» * %-C?^^ 



V-P 




>ov^" 







t* Vw ’ 


^ ffjT TV ^ ^ ^ ^ 9;;. 

I ^ o^fet 


c^^-> 

o 

-^6 ^ 
^ y^ r 



,0' cOX^*^ 

» Cj * ■•' 

0 “ “ 


8^ n ^ 

^ -» 
o 



^ ^ . V.=^ w . _ 





\A .. ^ <A. *>^0« ^cy ^ 


o 

"•J'i. y\ 

'*^> ^»ONO'5\<hV' 9^'•»,.. 

^ ^’^j. ^■%' ^'■^^^/v,%9(> A- 

-»^iyAx G-i- 


s<xs»**>-g 9> “'‘° ■^.1-'"* °*'Hx 





<5^S^ ^ 

O 


» ^ov 

y ^ <f‘ 

V "=>«<>’. 



* ’•■** ^toN 


«^"/V° ’ 




f ^ r^'^' ’>" X" 

>'^4^' ‘*'»i>N0’*v«^ 9). ^ 

c <( iV >. , ^v «* ■< ■^ O >, 

<r ^ Wf^iStet^'^ ^^'.’■!K\//l!’'r, 9f 

<t? 


o 




Vx 

^ A.^ 'C> ^ 

^ -V 




, x-»' .0"" '*& 

- A .ik 




i^oso’^ 


V % 

qv co«'=«.%r'''‘""i^ 

% 'Arf? « 

o ^ ^Q-y « ^ V'O^ o 

A *1 'S^Vii^ t«>9KtVr» Am-. <1 


^OV <» 

.o, -H, ‘>° .. * ^ ^\, ♦ o^Pv"*’ 

^ 'V>'6‘ o r c55'^'n 



cX-'^'-Xo 


^Q. 




C'"’ •ifcX'"'“:>X’, 


%X ,cp^ 

o 

t 

ibJ?, 



^OV « 


it TV 





X.^rxA yv* 

* ^v ^ 

%>^J.<9 X 9 




\ rfi 

' y* - v»s»<y- -, 

, »ono '5 

V^ 'S 


* 



> * ■« ^- A* 

*‘-'*/<f/“‘‘’-*o'$‘'^c,o«'=C 

% 'V cP * - 

; -f^d* "' 



’ * 
O 
to 







.4’^r. 



c,- 



























* LI, '^*■'0 , X'*'* -S^ ^0.\ 

<l'^ »* .■#J«>a'''. 'O 

V <-,^^; o 

- 1 X <» 





^ ov <» 







^ 1 ). s 




r>%. <iy -h- 

i \<^ “SSmt 



‘"-^'^"■\lw/ /’\ iW/ I 

6^'^^°’'“ v'V"" ’ ‘'"'►^V° ’ 

S^ * * -^o* ° g^ia,» '^ov «•* vd* ® 

‘ -'ft" tf j'M- ^^V-^'’-'' 

i>. a *■ < y 


O 






'Ta ^12® ^ o w/^w to ^ 


(!> 


^O, 



\o^' 


► .Tjv '-(;) .iT^ -Jv-r '1*^ /-V li 


'*' '^o> » 

.“V -— 

■^ov^ 


<P«C?3 C3 Kj 



% o 

O 


rCr f.oNG 



'o '.w.' o' X '.w." '•■ 

; ° "J%: %«v^ =mk\ ' 

r £5>J?n. 1 mi^ o .».VA - ,<?,J'^ n 


>5^'" V\W/ /\ %W/ 

>.<■ *•">/<>/“ • ’•'^ o^*^®'’"“«#'\ ^ 


’ONO'3^^ ^O.^it 


'^V ^ D. ^■^cy ° 

^ AN <* 


^1 ^ 
'o 

^ O 

V "<5*^ 

o <P c> 







tt'^ K^ 

, y n 
<« ^ 

’X'W/yV^- 

“ j» 1*^ -A^ ^ Avr/Vi^ ^ 

A^V!«Zrv^r. 

A 


^ <, A ^ono'^ C^ 

4*^-j^. .<v ^’■^^%(?/v, ^' 


: ipx ’- 

^ -O 


X'"’ ’i-*-” LiL'-V-“’ v<.< ■ »?■ 






%'\ ^^'•' 

* '^‘o’ ° , 

(O' .‘C'* V ■ ■‘^'’■^•1^*?'V”' *' d«'‘’V‘ • • 'J 

^ '^ov <*<0^: ^OV^ 

>P<i* ^ ^ ^ ^ 

. .. s - r V V °^"' 

*"o V 



ff n tv 


- t 



^ 'j'er ° 

* ° 

o 


^ »OKO'’ , *o»Pl' *»n*ViP 

* w :^m\ \c^ “"^fe: 






\6 

t % 


fo ^uyk ;« 

’>\^ * ofV'- * oS^ 

%• AV/h,’® '^V;<88tete\ A A'-'’^vA ^ 


>J 

'. 'ov* . 

w ’ 

, 4 / o 

* ^ s> 



% S 

* A\ \ 

' X . 0 ° 


■^fs- 


>.o« 




.Cs .0 































Medical Team Training—Technical Review 





Agency for Healthcare Research and Quality 
Advancing Excellence in Health Care • www.ahrq.gov 


PATIENT 

SAFETY 







This report may be used, in whole or in part, as the basis for development of clinical practice 
guidelines and other quality enhancement tools, or a basis for reimbursement and coverage 
policies. AHRQ or U.S. Department of Health and Human Services endorsement of such 
derivative products may not be stated or implied. 

AHRQ is the lead Federal agency charged with supporting research designed to improve the 
quality of health care, reduce its cost, address patient safety and medical errors, and broaden 
access to essential services. AHRQ sponsors and conducts research that provides evidence- 
based information on health care outcomes; quality; and cost, use, and access. The 
information helps health care decisionmakers—patients and clinicians, health system leaders, 
and policymakers—make more informed decisions and improve the quality of health care 
services. 




DoD Medical Team Training Programs: An 
Independent Case Study Analysis 


Prepared for: 

Agency for Healthcare Research & Quality (AHRQ) 

540 Gaither Road 
Rockville, MD 20850 
www.ahrq.gov 

Office of the Assistant Secretary of Defense/Health Affairs 
(TRICARE Management Activity) 

U.S. Department of Defense 
Skyline 5, Suite 810 
5111 Leesburg Pike 
Falls Church, VA 22041 
www.ha.osd.mil 


Contract No. 282-98-0029, Task Order No. 54 


Prepared by: ut 

American Institutes for Research 
David P. Baker, Ph.D. 

J. Matthew Beaubien, Ph.D. 

Amy K. Holtzman, M.A. 



OF 



AHRQ Publication No. 06-0001 
May 2006 


This document is in the public domain and may be used and reprinted without permission except 
those copyrighted materials noted, for which further reproduction is prohibited without the 
specific permission of the copyright holders. 

Suggested citation: 

Baker DP, Beaubien JM, Holtzman AK. DoD Medical Team Training Programs: An 
Independent Case Study Analysis. (Prepared by the American Institutes for Research under 
Contract No. 282-98-0029, Task Order No. 54) AHRQ Publication No. 06-0001. Rockville, MD: 
Agency for Healthcare Research and Quality. May 2006. 


♦ 





< 




11 


Acknowledgements 


This document was developed from work supported by the Agency for Healthcare Research 
and Quality (AHRQ) and the TRICARE Management Activity of the U.S. Department of 
Defense (DoD). The American Institutes for Research (AIR) would like to thank Dr. James 
Battles, Ms. Heidi King, ETC Beth Koshin, USAF, and CPT Glenn Merchant, USN, for their 
comments and advice during the preparation of this report. AIR would also like to express its 
thanks to the developers of MedTeams™, Dynamics Outcome Management®, and Medical Team 
Management for allowing us to attend their training and providing unlimited access to important 
training materials. Access to this information, in part, made this case study possible. We want to 
commend the efforts of Dynamics Research Corporation, Crew Training International, the U.S. 
Air Force, AHRQ, and the DoD for seeking to improve patient safety by promoting better 
teamwork in health care. We believe that efforts, like the ones described in this report, will 
ultimately lead to improved patient safety in the DoD and elsewhere. 


Ill 



\ ’ 


...... 




I. 


i -f 


^’1? V 


\i ' '■ . . ■■ ...r* .. -la pi 




i*^. l^•^-'..v. *A<^. >* 

I/I Ai»v*' ■'•'■- ' ■ ■ ... 

*...:,' r.n'-’ 








*<• 


•% r 


h 






■» . [: i\'i* ’ • »'• ’XCJJ 



.J^ 

r ' ‘V, 


-.! 





I * * *^, s 


* *■ 


tr * 



*M 


■ fA* 

■ I.'* i 




. ••’v.V'M'^ ; 

. i:'f''n;,v,,.....;.v-p',. 

• -<" k 

*. ' ■ i‘m v**^ 

A ^ 


■r. 



\A 

V •; a .'<• 


^... :' 

I* • * 


.'1 ■' '.I *"*c' 



i 


I 








Abstract 

Crew resource management (CRM), a form of team training that was developed for military 
(and, subsequently, civil) aviation has been adapted for use in health care settings. Several 
different CRM-derived medical team training programs have been in use within the Department 
of Defense’s health care system and in some civilian hospitals. The Agency for Healthcare 
Research and Quality awarded a contract to the American Institutes for Research (AIR) to 
evaluate three DoD-sponsored medical team training programs—MedTeams™, Medical Team 
Management (MTM), and Dynamic Outcomes Management® (now called LifeWings'^^). AIR 
had previously conducted a review for AHRQ of the research literature concerning medical team 
training. 

To conduct this evaluation, the researchers first reviewed student and instructor guides, 
slides, and other audiovisual materials that the course developers provided to them. Second, the 
researchers attended and observed the classroom portion of each of the three programs, collected 
pretraining data on student experiences and expectations and new, independent posttraining data 
on student reactions to the programs. Finally, trained AIR staff conducted one-on-one interviews 
with MedTeams and MTM instructors. 

The results suggest that all three programs possess several desirable characteristics, such as 
using active learning techniques and offering training to interdisciplinary health care teams. 
Nevertheless, each program also had a number of limitations. For example, not one of the 
programs was based on a comprehensive pretraining needs analysis and participants had limited 
opportunities to receive structured practice and feedback on critical teamwork skills. Based on 
their observations from the three case studies, the researchers set out the framework for a 
successful medical team training program, and recommended further actions to improve and 
support future medical team training programs. 


V 





>L^ . . .*, . 



* ■» >. 




-.__ < ■ ‘■•v^;*'It'.*?' i < ., ;, 

n.'lg '• 

..'r:.' : ■'*? 


j I** 





* Vj-i - ''tV., i'. *■ ■ :•• ■' 





.i-:-'jk t ,., . * 


I- 

"n*r ''Jf 

lyV^ir .’' 



■'':v:X-\-' *v'> 

l|*< • •» «S c 

•V , 






• . t -V 

. .. ,»\s^ /V . . 


.!r*^'>i*'' 

• I i 

l■^yl.^'' I.."; “Wi 


Al 

, t» 




■Kn^nr^H 


tk 

’':V (P^ 

'• I’-,),I'' ' * . ■ . ' 

***' .‘^.■•.r->* 



.s* ? ' 

,|r^‘V. vfl K.lv^^'v, til'.;.' . ■ '. / 



■ <• ■ 

,.» r, ■it...ilt... - * 

iJ-ilhiJW. • 

^ i. " * 






‘m~ ' >,.’. •■‘- V ■ i' 
t", * :.•: r, ' *■ 



^ .- . 'rfft' " 

• ■ >. * , . 
■»*'*■ ^ .r' 

■'I* 


J S 


, 

* i» ' . i , ^ •'ft';'* ' w ">'■ V ‘ 

^ • *^ ,»• * 

, 


i \ 

"* I 



■4 , ■ 


’1 


* Jf - '^■‘ 




lit 1 

.'I 


4 • 


• I ■ ■. fi’’ ''■■ 

■_ i% 


.'vLSi 


■/k. ■ 


1,, 


*•» . <• ■ ;1 *’- 
" , • • ♦ * t‘ 

1 -••4 ■ 

i-f . 

' .•* ' * * 

“ ’* J* 1 

't 


■ <','■''■1' ■ 

ip ■ 

■.,.o 

.« ♦ 

, lih* -Jf # 4 ’ J . 

■ f :-L-¥'^ . * '*. - j^ \m . ^ 

• .f ^• 

-V.-' 





) t' • ■ 
• ■. -V. 


< ■ ► 
M ", 


■ Aa3m 


. P. . .•• - .; , . 

^ „aV . r 











Contents 

Executive Summary.1 

Chapter 1. Introduction and Methodology.7 

Background.7 

Purpose of this Investigation..8 

Training Evaluation.9 

Instructional Systems Design.9 

Analytic Models of Health Care.10 

Case Study Approach.11 

Document and Literature Review.11 

Course Observations and Outcomes Reviews.11 

Description of the Participants.12 

Chapter 2. Case Study 1—^MedTeams*^^.15 

Introduction.15 

Case Study Approach.15 

Document and Literature Review...15 

Course Observations.15 

Pre- and Posttraining Assessment Tools.16 

Instructor Interviews.16 

Results.16 

Theoretical Basis.16 

Analysis of Training Needs.17 

Training Obj ectives.17 

Training Content.19 

Instructor Selection, Training, and Preparation.21 

Instructional Strategies.22 

Training Effectiveness.23 

Strengths and Weaknesses.26 

Summary.28 

Chapter 3. Case Study 2—^Medical Team Management.29 

Introduction.29 

Case Study Approach.29 

Document and Literature Review. 29 

Course Observations.29 

Pre- and Posttraining Assessment Tools.30 

Participant Interviews.30 

Results.30 

Theoretical Basis.30 

Analysis of Training Needs.30 

Training Objectives.31 

Training Content.31 

Instructor Selection, Training, and Preparation.32 

Instructional Strategies.32 

Training Effectiveness.33 

Strengths and Weaknesses.35 

• • 

Vll 














































Summary.35 

Chapter 4. Case Study 3—Dynamic Outcomes Management®.37 

Introduction.37 

Case Study Approach.37 

Document and Literature Review.37 

Course Observations.37 

Pre- and Posttraining Assessment Tools.37 

Instructor Interviews.38 

Results.38 

Theoretical Basis. 38 

Analysis of Training Needs.38 

Training Objectives.38 

Training Content.38 

Instructor Selection, Training, and Preparation.39 

Instructional Strategies.40 

Training Effectiveness.40 

Strengths and Weaknesses.42 

Summary. 43 

Chapter 5. Discussion.45 

Introduction.45 

Summary.45 

Are the Medical Treatment Facilities Ready for Training?.45 

Is Aviation Crew Resource Management the Right Starting Point for 

Medical Team Training?.45 

Have the Needs Analyses Gone Deep Enough?.46 

Are the Current Instructional Strategies Appropriate?.47 

Are the Current Methods for Delivering Classroom Training Sufficient?.47 

Are the Current Methods of Sustainment Sufficient?.48 

Recommendations.49 

Standardize the Knowledge, Skills, and Attitudes.49 

Identify Practice-Specific Training Requirements.50 

Leverage Existing Knowledge about Teamwork Training.50 

Develop a Standardized Training Specification.51 

Develop Advisory Circulars on Crew Resource Management 

Issues in Health Care.51 

Conclusions.52 

References.53 

Tables 

Table 1. Pretraining motivation.13 

Table 2. Organizational culture.14 

Table 3. Belief in the importance of teamwork.14 

Table 4. Summary of medical team training program content.18 

Table 5. Posttraining opinions about MedTeams'^^.24 

Table 6. Posttraining opinions about Medical Team Management.34 

Table 7. Posttraining opinions about Dynamic Outcomes Management®.42 


Vlll 












































Appendixes 

Appendix A: Pretraining Assessment Tool 
Appendix B: Posttraining Assessment Tool 
Appendix C: Instructor Interview Form 
Appendix D: Expert Panel Participants 

Note: Appendixes cited in this report are provided electronically at 
http://www.ahrq.qov/qual/teamtrain/index.html#app . 


IX 




I . ' 1 * 

1. .. ... , i, . 

'l.V. » ■ 

i)K‘ 


4 


ft 



f'? 


V 


« 




. I 


£75'' -i 

' ^ i 


( ' ♦li f 


.M- 


•r V> 

* ^ • 


.V 



V - . ^ ft jf 


^ - • rii - ■ li .. ■ li..'- ' , H 

. '•■■ '■•• , 4x ■ ' * t Ji 

^,, . ’5;W.v. ^ '• 

V ■ •'■■ .-I':'"! '’l^ » . ‘ 4:;,. * ‘ . ,’■**•»•■■ •'•’'^ ' 


.u'j; ■■Hi, 

* •+ ' ■ 

.«*‘-<j»' '*'k~ 

»V I “ 


.dsi«-4»*'rfW3''A 


•! 5 


. ^. ^ 


iu 


V t 





'fi* ^ 

■A^ -« ■ ■ • ' 

« 'i'V' _ _ 


•xv 

''a 








, V 






^ . .-* ’•j ’’ ^ .‘, • ‘ V 'T ' • t 

p,'- _ '■•<la«.'"v"' > ■ " t. •■•••." - 




'•.til''. ft. )(i> .1^- ^.. .M*,.).!; 1 . . ibr ... 


Jt — 





IW '¥ ‘4 i\ ■»■ < ■ 



'‘tJ.$y *-, •.* ,. f,. ' /* !V/' 

’ > , . / i. . , ■ ' _ 


f ( 

•JW' 




‘I?” '^'g/ ■’ 


■ ■■ ■ ..v;^ 


rr^ * > , . / i. j-v .,■■■_ ,. ‘1 • •'^- T / ■*' A’~1. ^ 

»--vSr ■ ■ ■■■■'V|?^^^^ ■• ^-,^V- , V,, 

|^ij^..,iT) .,_ :.,. . .. . ,>.A.,_. ^ ^ , et'**'- k''■^-W■■;;■ V'. . ■•'^ j 

(U’ ff -^"^It* 4t\-( > ^ -■’. >"f. • ■ '■•* , \ •.■ ■ ■ ’ ■■* .' >* . - K, J 

>7 i> ' ' *^y • I . •» ■ - *. ' ~ \A 

' '*• }i.tt.'h-\ ; ■ *■ ’ - -'^-"-.1-^. . i-.>'■ *. .** 'ild| 


^ 

^^.. ■ 

■'''»■*'" '■ . •■■ • ‘is®ws 





'y'^-\ ■ ')i 

*V 4Ai&:. .. ' ■' 


n-i 


?'V^ #''t 


. *• 


i./lf. 




. , '4- ■ i > ^ 

' -. .v-*rt , ihiM. i' * > ..i » m. ,'v " •#• 

j u/. ,... I'.i^ t L. ■ 4'ii-'" ; .♦ * !«■■' 






* ti 






*>-4 j .1 




(iiJfi'is*-fcw, 4^ 'A ^iiiHl!llB\'/i .... 

•'» ' ■ ‘* .'.j^ • f’*,-;.^ 'i^'V'''V.V^■'’'•. ^ 

I . ‘ ' 





. ^«k 


.1*** 


IS 

»*V ’ 


>f 






I V i 


- ' ■')-■■■ ■ -4 . .'^ ' ■» '"■., * 

it:' A' ..•uVc.’a* ^'Mk. IkM ..4 'iL'.!.^ ^(' 









Executive Summary 


Introduction 

For fiscal year 2003, the Office of the Assistant Secretary of Defense/Health Affairs tasked 
the Agency for Healthcare Research and Quality (AHRQ) to independently evaluate the best 
practices in Crew Resource Management (CRM)-derived training programs for military medical 
settings. The American Institutes for Research (AIR) was awarded a contract by AHRQ to 
conduct this research. 

This report describes the results of one of the major tasks performed under this effort, an 
independent case study analysis of Department of Defense (DoD)-sponsored medical team 

_ TrA A 

training (MTT) programs. Programs reviewed include MedTeams , Medical Team 
Management (MTM), and Dynamics Outcomes Management®(DOM).* The results described 
herein provide unbiased infomiation about each MTT program. Prior to this report, program 
developers had investigated their own MTT program’s effectiveness. Therefore this investigation 
was the first independent assessment. However, although this investigation was an independent 
assessment, the investigators only had limited access to certain kinds of information. Thus, they 
refrained from making judgments about the “goodness” or “badness” of each program. Rather 
the investigators provide comprehensive descriptive information. 

Approach 

Several sources of information were used when gathering data on MedTeams , MTM, and 
DOM. First, AIR investigators reviewed student and instructor guides, slides, and other audio¬ 
visual materials that course developers provided to them. Document review also included 
reviewing the relevant research studies that have been published on a specific program’s 
effectiveness. Second, AIR staff attended and observed the classroom portion of each of the 
courses. Furthermore, AIR collected pretraining data on student experiences and expectations 

... 'T’Xyf 

and new, independent posttraining data on student reactions to MedTeams , MTM, and DOM. 
Finally, for MedTeams and MTM, trained AIR staff conducted one-on-one interviews with 
instructors who had taught or were going to teach the course. 

Results 

The results suggested MedTeams , MTM, and DOM possess several desirable 
characteristics, such as using active learning techniques to develop the participant teamwork- 
related competencies and offering interdisciplinary training to teams of physicians, nurses, 
technicians, and other heath care professionals. Nevertheless, each program also had a number of 
limitations. For example, none of the programs were based on a comprehensive pretraining needs 
analysis; limited opportunities existed for participants to receive structured practice and feedback 
on critical teamwork skills; and few strategies were available for sustaining and reinforcing 
teamwork principles in the posttraining environment. 


* Dynamic Outcomes Management has been renamed LifeWings^'^. 


1 




Conclusions 

Based on the findings from this investigation and other tasks performed under this contract, 
AIR defines a successful medical team training program as including the following 
characteristics: 

• A comprehensive pretraining needs analysis that identifies an organization’s readiness for 
change, possible barriers to training transfer, and common and unique team requirements 
as a function of medical specialty area. 

• A set of validated team knowledge, skill, and attitude competencies that were identified 
as important during the needs analysis. 

• Awareness training that highlights the importance of human factors principles in 
medicine and develops important team-related knowledge competencies. 

• Multiple opportunities for trainees to receive structured practice and feedback on specific 
team skills through simulation, on-the-job training, or some combination of the two. 

• Recurrent training that reinforces team knowledge and skills to prevent skill decay over 
time. 

Recommendations 

In addition to defining the key features of a successful medical team training program, the 
investigators made a number of recommendations to advance both the practice and science of 
MTT. These recommendations are as follows. 

• First, they recommend that the DoD develop a standard list of generic teamwork-related 
knowledge, skill, and attitude competencies that represent the core elements of successful 
team\vork in health care. 

• Second, the investigators recommend that DoD and AHRQ identify how team 
knowledge, skill, and attitude competency requirements vary by medical practice, acuity 
or other job characteristics. Identification of such competencies would lead to practice- 
specific team requirements and tailored MTT for specific disciplines. 

• Third, they recommend that instructional designers look beyond aviation CRM training 
and leverage all available research and tools (i.e., a tremendous amount of research on 
teams has been conducted by the U.S. Navy) when developing medical team training 
programs. 

• Fourth, the investigators recommend that the DoD develop a detailed training 
specification for MTT programs for awareness, practice and feedback, and recurrent 
training. In addition to describing the core teamwork competencies, the specification 
would outline the appropriate instructional strategies for each core competency, the 
appropriate sequencing of training activities, and outcome measures for assessing the 
degree of skill acquisition. 


2 


• Finally, they recommend that AHRQ develop technical assistance documents under the 
Patient Safety and Quality Improvement Act of 2005 (P.L. 109-41) on issues related to 
team training and error prevention, much like the FAA’s advisory circulars. The 
investigators believe that human factors-related advisory circulars would go a long way 
to educate the medical community about the importance of MTT for ensuring patient 
safety and for ensuring consistency across MTT programs. 

Summary 

In conclusion, this report presents an in-depth case study analysis of MedTeams , Medical 
Team Management, and Dynamic Outcomes Management®. This was the first independent 
assessment of these programs. The case study approach allowed for the collection of detailed, 
comprehensive information on each program, which was reported along a common set of 
variables. 


3 




. » i - • ! ■ 1 

,-r f i; -.M/ y. , 





■' * - ’ " J ■ :>■. 


Vwt;, ('^« ■> ’'^ 

"■;u> '. twT 


fe^>5 k^ttwict 1<5 
H§»»:i; .-W-WKW- 


n* 



ipfyifj'<»»''<'' 

. ..I'Y-r iv ■*:... • • 


I'|T .' ' V>.w IV 

f<7?y • I,' * ;* 

« 







_•>' r 

• />■ ■ "> ft 


u V 


/ a 
>• 


J 




.» 


s ‘ 


yn 


f, .*».. 


j • # .f' ■ ’'■•' • ■. 

r- ' ’ * * ^ 

' ■. X. ^ 


.,: .-H 

ii'^*'':.2 A .■/i'filji- \', i! 1? 




<» 


' ei 







cnX^V ■i-.. -t^A. ■'‘^^'f'l,''’'f-fv<^* 


















DoD Medical Team Training Programs: An 
Independent Case Study Analysis 



vf 





- " ■ 





% 




tv M r*j ' 

\' »i . • 

% 


’V- •-''../A 

••.. . \ I ' 


» •: *■.-!» •' 

■I ", -f: 








•■" ,r 


***IH''V t!. * * 


,■ I • ■ 


,. I 


•-; 'r *. ■ „ 


V, ., . & '?r 


<»V' 


*4 M 










■i . k. * 'V" 

^ *.1 ■*: T ■•■ ■*' -*=■■ 


r '^4^’ ' V •' . 

.. ft , ' 


?• *• . 1 I 

■ ■’■■ 

' -.fj 

'J S ■>!*?« 


,) 






♦'•y - • . I -*■•*. 

*.f I > 



tt ’;■• r ,’ _ 4 . w 

Kv. t 

V, 


tiJi'A .-.k . 

[ ' • ' ■ 

• j • ^ »• 



►i4ifr jm .,, \ ..•••. 

r-£: ' •' ‘ '' 

T, /)'•“ ■ ' t I ■’ ' ',sl ,' - V 

* ■ 'i a,‘.■. *' • '. ' 't , 'I'’''i'^-''V'' v 

■.'• ... V ;•• ■v'^ 

’ ■^Av.„.W / f^ •' ■ ■•'^ ' *-.^5 

.. * , 1 , _ •. -' ■-' ■■■ .. ■* A -'ill. 






.' JV 

‘‘■‘hiJ >iw -r-a ' -»* 


-('^£,?-l 

J*I ■ • 


-iff-' '■'. 'll , • ■'■ 

'* •*'*■ ' *> -I I 

■ V •’ " 'V :' i 



*5. 


i'V 'sj^v j‘M ,>u • 

'(• '-..I ‘ H' ■ iV-'‘w • ^ ‘a"' . ', . 

- ' '' * * * . 'X* ■ '' 

■'■ 

L*-' ' ■"'‘<*-v 





I ‘ > , ■'» A'*.'" ^“. 

’, ’ ■ ' 

■' ■ r,' • 


.■«‘'F^^-.', it 


' ^ 

if ' 

I. • • «. 

. r. ”• '♦ 

' -ic- ■ ■■ 







. <_ 








:a 


; ¥ f)m-rS:W ' W -:f, <i«;f 

f it«J..'‘-'I- . ' 

w, . - ■ n. .. 






f- ■.,»/ /•'» ^ 



VH < 




=? ilV'-'V “ ' 

iXi •* ^ 







>'■1 


.r'f 


• -w > 






,t 


i.' ^ r 




.A- 




- t .-•' ’• * ih 


■V 


<•*/ r 


j,-: 

4 -uLlAJii. LLf-l-V’ U ->'\ 


j ■) 


;A 


/ ‘if 




s* i 


r.fr'' I - ..v„ 

^': f ■'I 11 |‘■\^ " ,,f I •,'■ /' 

■'.‘•'‘.i. :,0-LiA r 


■■• i ‘I' 


i»' 


V H ■• -'V-‘» * , f 


y II /.', )', 


*' * )„-'■ ’ 

' , > • 

‘ V, 






,y I 


..... ^ ^ » 
[Si!>'47'(’ 







# , I 


A* 







/ 

» f 


r . 







LUJi' • ft-. 1 .'-^ 


' 1 . .•, * < '. 

I ' '-W 

. Ik kV.. 











Chapter 1. Introduction and Methodology* 

Background 

Patient safety is a top priority in health care. The Institute of Medicine’s (lOM) publication. 
To Err is Human: Building a Safer Health System,^ concluded that medical errors cause up to 
98,000 deaths annually. The lOM report brought national focus to this important issue, and has 
now spawned significant research on the underlying causes of medical errors and the 
effectiveness of different strategies for improving patient safety. Although still in its infancy, 
research on strategies to improve patient safety appears to have a bright future. 

The Quality Interagency Coordination Task Force (QuIC) was established in 1998 to address 
a number of critical needs identified in the lOM report. The QuIC is composed of representatives 
from different Federal agencies including the Departments of Health and Human Services 
(HHS), Labor (DoL), Defense (DoD), and Veterans Affairs (VA), to name a few. Currently, the 
QuIC has completed work on existing projects and is awaiting input from members on future 
initiatives to pursue. The QuIC identified 100 activities for improving patient safety, most of 
which have been implemented by Federal health care organizations. Among the QuIC’s 
recommendations is the widespread adoption of human factors-based training, such as Crew 
Resource Management (CRM) training, for improving teamwork in health care. The QuIC 
believes that lessons learned from high-risk environments should be looked at when developing 
new patient safety practices. 

Helmreich and Foushee concluded that the introduction of CRM has been one of the greatest 
success stories in aviation. The efficacy of CRM has been established, in part, because CRM has 
been evaluated throughout its evolution. CRM training was developed interactively—introducing 
and testing the effectiveness of different strategies, which allowed for the best possible results.^ 
CRM training is the gold standard for what can be produced when those interested in research 
and those interested in practice work cooperatively to achieve a common goal. 

The Federal Aviation Administration (FAA) has issued specific guidance on the development 
and conduct of CRM training.^^ The FAA describes CRM in three phases: an awareness phase, a 
practice and feedback phase, and a continual reinforcement phase. The first stage of CRM 
training, awareness training, involves communicating teamwork principles and concepts that are 
fundamental to a particular task domain and developing attitudes and beliefs that will motivate 
trainees to be receptive to those ideas."^’^ The second stage of CRM training, skills practice and 
feedback, involves developing the skills necessary to apply the concepts that were introduced in 
the awareness stage to on-the-job situations. This stage is critical because “individuals may 
accept, in principle, abstract ideas of [crew resource management concepts] but may find it 
difficult to translate them into behavior” on-the-job.^ The final stage of CRM training, continual 
reinforcement, involves repeated exposure to CRM concepts, as well as on-the-job feedback and 
reinforcement of CRM concepts from multiple sources. 

Perceived parallels between health care and aviation have led to a number of CRM-derived 
medical team training programs. Applications of CRM in medicine started with the introduction 
of Anesthesia Crisis Resource Management (ACRM) at Stanford University School of Medicine 


* Note: Appendixes cited in this report are provided electronically at 
http://www.ahrq■aov/qu3l/teamtrain/index.html#app . 

^ The current version is Advisory Circular AC 120-5IE, dated Jan. 22, 2004. 


7 





and the Anesthesiology Service at the Palo Alto (CA) Veteran Affairs Medical Center.^ More 
recently, DoD has funded several medical team training initiatives. MedTeams has been 
implemented in Army and Navy hospitals, while Medical Team Management (MTM) has been 
introduced in the U.S. Air Force.^ However, these training programs have not undergone the 
same scrutiny as CRM. Some evaluation studies have been conducted on ACRM, MedTeams™, 
and MTM, but not with sufficient rigor to draw firm conclusions about each program’s 
effectiveness or the relative effectiveness of different training strategies for promoting teamwork 
in health care and reducing negative patient outcomes. 

The importance of training evaluation is unquestioned by instructional designers and training 
researchers; however, training is often designed and developed but not evaluated. Training 
evaluation has been defined as “the systematic collection of descriptive and judgmental 
information necessary to make effective decisions related to ... various instructional activities.”^ 
These decisions include determining whether the goals and objectives of a program are 
appropriate to achieve the desired outcome, whether the content and methods in training will 
result in achievement of the overall program goals, and how to maximize training transfer. 
Although systematic training evaluation is not an easy task, it is the only way to ensure that 
training programs have the desired effect and are a worthwhile investment for an organization. 

Purpose of this Investigation 

The purpose of this investigation is to conduct a case study review of DoD-sponsored 
medical team training (MTT) programs. The American Institutes of Research (Washington, DC) 
conducted the investigation under contract to AHRQ during the summer of 2003. Programs 

TT'Xyf 

reviewed include MedTeams , Medical Team Management (MTM), and Dynamics Outcomes 
Management® (DOM).^ 

A case study approach was selected for this initial investigation because we did not have the 
access or the resources to conduct an empirically sound, comparative analysis. For example, 
convenience samples were relied upon when collecting data from program participants. This 

*T*Xyf 

resulted in data being collected fi-om MedTeams participants from a DoD facility, future MTM 
instructors, and DOM participants fi*om a university medical center. Moreover, we only had 
access to the classroom (awareness phase) portion of each training program. All of the programs 
address the need for skills practice and feedback through a variety of post-classroom activities, 
however, due to time constraints associated with this contract, we did not have an opportunity to 
observe these actions. Therefore, it is important to keep in mind several key characteristics of 
this investigation when reviewing this report. First, this report describes case studies of three 
existing medical team training programs: MedTeams , MTM, and DOM. Although data for 
each case study were described along the same set of dimensions, this information is not 
statistically comparable because it was derived from different information sources with different 
levels of rigor. Second, the goal of this report was to provide unbiased information to the DoD 
and AHRQ about these programs as a whole. The program developers have conducted past 
investigations, and therefore this investigation was the first independent assessment. Finally, 
because we only had access to certain kinds of information, we refrain from making judgments 
about the “goodness” or “badness” of each program. Rather, we provide comprehensive 
descriptive information. 


^ Dynamic Outcomes Management has been renamed LifeWings™. 


8 



Prior to describing each program, we first present an overview of training evaluation. Here, 
we will describe the prevailing approaches for determining training effectiveness, both within 
and outside of health care. Different approaches exist and we will review the relative merits of 
each. Second, we present our case study approach for MedTeams™, MTM, and DOM. To the 
extent possible we relied upon best practices from the literature to develop our approach. 
However, we had to adapt many of these practices to the unique characteristics of the current 
situation. In this study, we were highly constrained by the extent to which we could gain access 
to the participants who completed each training program. Our case study approach to 
investigating these programs emerged from these constraints. Finally, we describe the 
individuals who participated in MedTeams™, MTM, and DOM. For each participant we 
collected information on several critical pretraining factors that can influence motivation to learn 
during training. 


Training Evaiuation 

Training evaluation is perhaps one of the most difficult, yet most important, activities in the 
instructional development process. The purpose of training evaluation is to systematically collect 
data about a training program to determine the program’s overall effectiveness.^ Evaluation can 
be used to determine whether people liked the training program, whether they learned what was 
intended for them to learn, and whether it positively affected their job performance.^^ It can also 
be used to gather information about specific aspects of a training strategy such as the usefulness 
of training materials (e.g., manuals and videotapes), attitudes about new technologies (e.g., 
automation) and instructional techniques (e.g., lecture versus practice and feedback). When 
multiple training programs are evaluated on the same dimensions, training programs can be 
compared to one another, as can different groups of trainees. 

A number of approaches have been proposed for conducting detailed training evaluation 
within different disciplines. Below we review evaluation strategies from the domains of 
instructional systems design and health care. Then we describe the approach that guided our case 
studies of MedTeams™, MTM, and DOM. 

Instructional Systems Design 

Kirkpatrick^^ proposed one of the first and still most influential models of training evaluation 
in which he outlined four types (Level I, Level II, Level III, and Level IV) of outcome measures: 
reactions, learning, behavior, and results. 

• Reactions (Level I) are simply measures of how well trainees liked or valued a training 
program and certain aspects of the program. 

• Learning (Level II) involves measuring the extent to which trainees understand and retain 
principles, facts, and techniques that are imparted during training. 

• Behavior (Level III) refers to measures of any behavioral changes that occur as a result of 
training. 

• Results (Level IV) refer to the measurement of the impact of training on organizational 
criteria; criteria such as increased profitability or enhanced customer satisfaction would 
be indications of training effectiveness. 

A review of the training literature suggests that Kirkpatrick’s Level I data, trainee reactions, 
are the most commonly collected outcome when evaluating training. Reaction measures focus 


9 


1 ^ 

on a trainee’s affective reactions to training, under the assumption that the extent to which 
trainees liked the training is correlated with performance on the job. Research that has explored 
the relationship between trainee reactions and other training outcomes, however, has found that 
trainee affective reactions correlate only slightly with learning and on-the-job performance (i.e., 
behavior). ’ Thus, simply asking trainees whether they like a training program does not 
necessarily predict whether they actually learn anything or alter their behavior on the job. 
Nonetheless, affective reactions are important. Trainees are the customers of training and 
negative reactions to training can undermine the credibility of training and the sponsoring 
organization. 

Recent research has indicated that asking trainees about the instrumentality or utility of 

training is preferred to simply asking them whether or not they liked the training. An example of 

this type of question might be to ask trainees to indicate the extent to which they perceived 

training to be of practical value. Such questions seek to determine the usefulness of training for 

1 ^ 1/1 

performing a job. Alliger and his colleagues ’ found that, although affective reactions to 
training (e.g., “Training was enjoyable.”) did not relate to subsequent job performance, utility 
reactions (e.g., “Training was useful.”) were related to both learning and performance on the job. 
Interestingly, these researchers found that measures of learning correlated less with on-the-job 
performance than either utility reactions or the combination of affective and utility reactions. 

In addition to the recent advancements made by Alliger and his colleagues, other researchers 
have made contributions to the domain of training evaluation by expanding the Kirkpatrick 
model. Kraiger and his colleagues advocated a multidimensional view of Kirkpatrick’s Level II 
data. These researchers decomposed learning into cognitive, affective, and skill-based outcomes. 
Similarly, Kraiger and Jung^^ suggested several processes by which learning outcomes can be 
derived from and linked to instructional objectives of training. Finally, Baldwin and Magjuka^^ 
pointed out that outcomes are influenced by a number of antecedent conditions that exist and 
interact within an organization. These include the manner in which training is introduced 
(voluntary/mandatory, goals, etc.), trainee past experience, pretraining motivation, and the extent 
to which a positive transfer climate exists within the organization. For example, mandated 
training will be successful when employees’ past experience with similar, mandated programs 
has been positive, leadership endorses the importance of the training program, and new 
knowledge and skills are reinforced on the job. 

Analytic Models of Health Care 

Donabedian’s structure-process-outcome model serves as a unifying framework for 
examining health services and assessing patient outcomes.Donabedian defined structure as the 
physical and organizational properties of the settings in which care is provided,as the 
actual treatments and procedures that are done for patients, and outcomes as what is actually 
accomplished for patients. From the standpoint of patient safety, Donabedian’s model provides a 
framework for examining how risks and hazards that are embedded within the structure and 
process of care have the potential to cause injury or harm to patients. For example, individual or 
team failures on the part of health care providers have been consistently cited as leading to 
negative patient outcomes. 

1 o 

Coyle and Battles modified the Donabedian model to include important antecedent 
conditions that can affect patient outcomes. These researchers suggested that accounting for 
environmental and patient factors is critical in understanding the effectiveness of any new 
strategy that is introduced or modifications that are made to the patient care process. In health 


10 


care, improving patient outcomes is the ultimate criterion for a strategy to be deemed successful. 
A change in process must lead to a corresponding change in patient outcomes. Under patient 
factors, Coyle and Battles included genetics, sociodemographics, health habits, beliefs and 
attitudes, and preferences; under environmental factors they included cultural, social, political, 
personal, physical, and other factors related to the health professions. 

Case Study Approach 

For each training program, we were able to access several sources of information. First, we 
reviewed all available documentation. Document review included reviewing all student and 
instructor guides, slides, and other audio-visual materials that course developers provided to 
AIR. Document review also included reviewing the relevant research studies that have been 
published on a specific program’s effectiveness. In addition to our document review, we 
observed the classroom portion of each of the courses. Furthermore, for each course we 
observed, we collected pretraining data on student experiences and expectations and new, 
independent posttraining data on student reactions to MedTeams , MTM, and DOM. Finally, 

*TTV>f 

for MedTeams and MTM, we conducted one-on-one interviews with instmctors who had 
taught or were going to teach the course. Each of these activities is briefly described in more 
detail below. 

Document and Literature Review 

As part of a larger state-of-the art literature review, AIR reviewed the available research on 
medical team training and specific medical team training programs. This review identified eight 
articles, conference papers, or technical reports published on the MedTeams approach and one 
article describing the effectiveness of DOM. Key to our assessment of MedTeams was an 
article published by Morey and colleagues^ ^ that describes an evaluation of the Emergency 
Department (ED) curriculum. Key to our assessment of DOM was an article published by Rivers 
and colleagues^^ that describes an evaluation of DOM training. No comparable investigations 
were found for MTM. Results of these papers are described when we review additional evidence 
that supports each medical team training curriculum. 

In addition to reviewing the literature, we carefully reviewed the instructor and student 
guides. For MedTeams™, we reviewed the instmctor’s guide for the Labor & Delivery (L&D) 
Team Coordination Course and the student’s guide for the Operating Room (OR) Team 
Coordination Course. For DOM, we reviewed the student’s guide for a course conducted at a 
university medical center. Finally, we reviewed the instmctor’s guide for MTM. Course 
developers provided these guides to us when we attended and observed each class. 

Course Observations and Outcome Reviews 

Course observations. AIR staff observed at least one session of the classroom portion of 
each of the three training curricula. For MedTeams , we observed the train-the-trainer portion 
of the L&D Team Coordination Course during fall 2002 in Boston, MA, and 11 sessions of the 
OR Team Coordination Course at a U.S. naval medical center during summer 2003. For MTM, 
we observed the train-the-trainer course that was conducted during summer 2003 at a U.S. Air 
Force base. Finally, for DOM, we observed a shortened version of the curriculum that was 
sponsored by the State Volunteer Mutual Insurance Company (Brentwood, TN) and two full- 
length sessions of the course that were conducted at a university’s regional medical center. 


11 


Pre- and posttraining assessment tools. For each of the courses that were observed (except 
the MedTeams™ L&D Team Coordination Course and the shortened version of DOM), AIR 
staff administered pre- and posttraining assessment tools. The pretraining tool was designed to 
assess student experiences prior to training and gauge participant attitudes and beliefs about 
teamwork (see Appendix A). Pretraining experiences and participant attitudes have been shown 
to affect trainee motivation to learn and reactions to training. ’ The posttraining tool measured 
participant affective and utility reactions to training (see Appendix B). 

A 

Instructor interviews. During our observations of MedTeams and MTM, we also 

conducted instmctor interviews. With respect to MedTeams , we interviewed 14 hospital staff 

TTVl 

members during the week of June 9-11,2003, who served as instructors in a MedTeams 
program that week. With respect to MTM, we interviewed 10 individuals who were attending or 
had previously attended a train-the-trainer course. In all cases, instructors were asked about how 
they were selected, their background in training, and perceptions of the course and training aids 
used (see Appendix C). Interviews were not conducted with the two DOM instructors, because, 
unlike MedTeams and MTM, DOM does not utilize a train-the-trainer strategy. The course 
developer. Crew Training International (CTI), provides instructors for all administration of DOM 
training. 

Description of the Participants 

AIR staff interviewed 14 instructors, and collected pre- and posttraining assessments on 223 

A 

MedTeams participants; interviewed 10 participants and former participants, and collected 
pre- and posttraining assessments on 26 MTM participants; and collected 78 pre- and post¬ 
training assessments from DOM participants. Below we describe our findings from the pre¬ 
training assessment, which were used to determine if participants were similar across 
MedTeams™, MTM, and DOM training. 

The pretraining results highlight some interesting similarities among the three groups of 
respondents (see Table 1). We began by reviewing their motivation to participate in medical 
team training. With the exception of one question (Question #3), nearly 50 percent or more of all 
participants had personally witnessed a breakdown in teamwork that could have compromised 
patient safety. In all cases, the two most frequently cited problems involved not learning from 
prior mistakes (Question #4), and poor preparation (Question #5). Fortunately, the participants 
did not report feeling pressured to perform procedures that they were not comfortable doing 
(Question #3). To empirically assess the similarity across the MedTeams™, MTM, and DOM 
participants, we ranked the percentage of agreement responses (within each program) and 
calculated the mean correlation across the three programs, using Spearman’s correlation for 
ranked data (rg). The average correlation was 0.69, indicating a moderate degree of similarity 
among participants. 

We found similar results with regard to the participants’ respective organizational cultures. In 
all cases, over 50 percent of the participants in each program agreed that their respective 
organizations promoted a positive safety culture (see Table 2). Participants in all three programs 
generally agreed that disruptions in patient care were the greatest detriments to patient safety 
(Question #8). However, they were uniformly less likely to agree that team members in their 
departments know each others’ responsibilities (Question #9). Again, we calculated the mean 
correlation across the three programs, using Spearman’s correlation for ranked data (rs). The 
average correlation was 0.46, which is somewhat lower than their pretraining motivation levels. 
This was not unexpected, given that each facility is expected to have its own unique culture. 


12 


Table 1. Pretraining motivation (% “yes” responses) 


Survey item 

MedTeams™ 

(n=223) 

MTM 

{n=26) 

DOM 

(n=77) 

1. Have you ever worked in a medical team where you 
did not feel comfortable voicing your professional 
opinion? 

46% 

58% 

68% 

2. Have you ever worked in a medical team where 
there was no clearly designated leader? 

48% 

46% 

64% 

3. Have you ever felt pressured to perform a medical 
procedure that you felt uncomfortable doing? 

33% 

46% 

56% 

4. Have you ever witnessed an actual or potential 
mishap that could have been prevented if the team 
had learned from their previous mistakes? 

58% 

63% 

86% 

5. Have you ever witnessed a routine medical 
procedure that went wrong because the team did not 
adequately prepare beforehand? 

60% 

50% 

77% 


Finally, we reviewed the participants’ belief in the importance of teamwork. The participants 
uniformly agreed that teamwork was important for ensuring patient safety. In fact, the percentage 
agreement for these items was generally greater than 80 percent and never went below 73 
percent (see Table 3). Prior to participating in their respective MTT programs, the participants 
were uniformly confident in their ability to work effectively in a team environment (Question 
#12) and in their belief that people with strong teamwork skills are likely to be successful in 
health care (question #14). Again, we calculated the mean correlation among the three programs 
using Spearman’s correlation for ranked data (rs). The average correlation was .73, indicating a 
moderate degree of similarity among participants. 

In summary, although the results did differ slightly, there was a common pattern of responses 
across MedTeams™, MTM, and DOM participants. Regardless of the institution in which 
training was delivered, nearly half of all participants had witnessed a breakdown in teamwork 
that could have compromised patient safety (see Table 1). Despite this, many felt that their 
organizations exhibited elements of positive safety culture (see Table 2), and that teamwork 
skills are important for maintaining patient safety (see Table 3). Therefore, we conclude that 
participants in MedTeams™, MTM, and DOM possessed similar motivation to participate in 
training and learn the material. With that in mind, we present our case studies of MedTeams™, 
MTM, and DOM. 


13 












Table 2. Organizational culture (% agreement) 


Survey Item 

MedTeams™ 

(n=223) 

MTM 

(n=26) 

DOM 

(n=77) 

6. The culture in our department makes it easy to 
learn from the mistakes of others. 

68% 

81% 

51% 

7. Our doctors, nurses, enlisted personnel, and other 
team members work together as a well-coordinated 
team. 

57% 

52% 

63% 

8. Disruptions in patient care can be detrimental to 
patient safety. 

76% 

88% 

69% 

9. Physicians, nurses, enlisted personnel, and other 
team members in this department know and 
understand each others’ respective responsibilities. 

53% 

52% 

54% 

10. My department does a good job of training new 
personnel. 

53% 

58% 

51% 


Table 3. Belief in the importance of teamwork (% agreement) 


Survey Item 

MedTeams^^ 

MTM 

DOM 


(n=223) 

(n=26) 

(n=77) 

11. Teamwork deserves more attention In health care. 

81% 

92% 

97% 

12. lam confident about my ability to work effectively 
in a team. 

97% 

96% 

96% 

13. Teamwork is one of the most important skills in 
the operating room (OR). 

92% 

92% 

87% 

14. People with strong teamwork skills are more likely 
to be successful in health care. 

94% 

100% 

99% 

15. It is impossible to function in health care without 
being a good team player. 

73% 

77% 

81% 


14 























Chapter 2. Case Study 1—MedTeams™* 

Introduction 

The primary purpose of MedTeams is to reduce medical errors through interdisciplinary 
teamwork. MedTeams™ was developed by Dynamics Research Corporation (DRC), of 
Andover, MA, on the premise that most errors result from breakdowns in systems-level defenses 
that occur over time. According to the MedTeams curriculum, each team member has a 
vested interest in maintaining patient safety and is expected to take an assertive role in breaking 
the chain of events leading to an error. MedTeams™ defines a core team as a group of 3-10 
(average = 6) medical personnel who work interdependently during a shift and who have been 
trained to use specific teamwork behaviors to coordinate their clinical interactions. 

Case Study Approach 

Consistent with our plan, we were able to access several sources of information to evaluate 

T*X >f TTi H 

MedTeams training. Specially, we reviewed all course materials for both the MedTeams 
Instructor Certification Course for Labor & Delivery (L&D) and the student guides for the 
Operating Room (OR) Course. We also reviewed the relevant research studies that have been 

_ TTVyf 

published on the effectiveness of MedTeams training. In addition to our document review, we 
observed 1 day of the L&D train-the-trainer course and multiple sessions of the OR student 
course. At the OR course, we also were able to conduct a pretraining assessment of student 
experiences and expectations, a posttraining assessment of student reactions to MedTeams , 
and interviews with 14 instructors. Each of these activities is briefly described in more detail 
below. 

Document and Literature Review 

Our document review identified eight articles, conference papers or technical reports 

Tr*\ K Tnv >f 

published on the MedTeams approach. Key to our assessment of MedTeams training was a 

paper published by Morey and colleagues^^ describing a quasi-experimental evaluation of the 
effectiveness of this training. Results from this paper are described when we review the 

_ 'TTVyf 

additional evidence that supports the MedTeams approach. 

In addition to reviewing the literature, we carefully reviewed the instructor and student 
guides from the L&D Team Coordination Course and the OR Team Coordination Course, 
respectively. DRC staff provided these guides to us when we attended and observed each of 
these curriculums. 

Course Observations 

AIR staff observed the train-the-trainer portion of the L&D Team Coordination Course 
during fall 2002 in Boston, MA. This course was taught by DRC as part of the large, on-going 
course validation study. A variety of staff attended this training from hospitals randomly 
assigned to the experimental condition for the study. AIR staff observed 1 day of a multiday 
course, which is designed to certify hospital staff as MedTeams™ instructors (as well as meet 
the requirements for participation in the L&D study). The observed day involved DRC 


* Note: Appendixes cited in this report are provided electronically at 

http://www.ahrq.eov/qual/teamtrain/index.htinl#app . 


15 






instructors providing the MedTeams classroom training to the future instructors. Essentially, 
the purpose of this day was to demonstrate how the classroom portion of MedTeams™ was to be 
conducted. On a subsequent day, the instructor trainees would practice delivering this training. 
This session was not observed. 

In addition to the L&D course, AIR staff observed the OR Team Coordination Course at a 
U.S. naval medical center during summer 2003. Twelve classes were conducted over a 1-week 
period, including two for the ophthalmology service, one for the dental service, three for the 
orthopedic service, one for the urology service, one for plastic surgery, two for general surgery, 
one for the cardiothoracic service, and one for the neurology group. Hospital staff who had 
previously completed the MedTeams instructor certification course and had been certified as 
MedTeams instructors taught these courses. We observed the didactic portion of this training. 

Pre- and Posttraining Assessment Tools 

In addition to observing the courses, AIR administered pre- and posttraining assessment tools 
to 223 participants in MedTeams training. The pretraining assessment tool was designed to 
assess student experiences prior to training and gauge participant attitudes and beliefs about 
teamwork. The results from this measure were described in an earlier section of this report 
entitled, “Description of the Participants” (p. 12-14). The posttraining assessment tool measured 
participant utility reactions to MedTeams training. All attendees completed the pretraining 
tool, but only 218 completed the posttraining tool. Of the 223 students, 94 were physicians, 
dentists, or oral surgeons; 62 were corpsmen; 44 were nurses; and 23 represented other groups, 
such as surgical technicians. 

Instructor Interviews 

AIR staff also interviewed 14 hospital staff who conducted the observed MedTeams™ 
training. Interviews were conducted with trained instructors from the cardiothoracic, dentistry, 
general surgery, neurology, ophthalmology, orthopedics, plastic surgery, and urology groups. 

The pool of instructors included department heads, physicians, nurses, and surgical technicians. 

'TTiyf 

Instructors were asked about how they were selected to teach MedTeams , their training 
background, and the flow of the course and training aids used (see Appendix C for the Instructor 
Interview Form). 

Below, we present the results from our analysis of this information. These results are 
organized around a set of variables that we used to draw conclusions about each training 
program. For each variable we provide descriptive data from our literature review and then 
findings from our observations, data collections, and interviews when appropriate. Table 4 
provides a summary of this information for each of the three training programs reviewed. 

Results 

Theoretical Basis 

The core objectives, curriculum, and instructional strategies for initial MedTeams™ tr ainin g 
for emergency departments (EDs) were derived from a CRM training program that was 
originally developed for U.S. Army helicopter crews to train them in specific behavioral 
skills.^^’^'^ The MedTeams™ developers argued that emergency medicine and aviation share a 
number of similarities, making CRM training for helicopter crews portable to the ED. These 
similarities include: the need for decisionmaking based on incomplete or conflicting information; 
the demand for coordination among professionals with varied skills and ranks; and the possibility 


16 


of poor team performance leading to serious consequences or death. Since the development of 
the emergency department course, MedTeams™ training has been developed for the L&D 
Service and the OR. These courses are based on the initial emergency department (ED) course, 
but case studies and examples have been modified to make MedTeams™ contextually 
appropriate for the service in which it is implemented. 

Analysis of Training Needs 

As part of the initial MedTeams development process for the ED, DRC conducted an 
analysis of team performance deficiencies. They analyzed the closed-case files from eight 
hospitals that were collected over a period of several years. Each file was then classified using a 
teamwork failure checklist to identify trends. Their analyses identified approximately 8.8 
teamwork failures per closed case. Based on their analyses, DRC concluded that improved 
teamwork could have saved the hospitals approximately $3.50 per ED patient visit. By 
way of comparison, malpractice costs range between $2-$6 per patient. 

TTlVI 

The initial version of MedTeams was developed using an evaluation-driven course design. 
Based on the closed-case file review, DRC identified five critical teamwork dimensions that 
were necessary for effective teamwork. They then identified 48 specific, observable behaviors 
that were linked to these dimensions, and developed Behaviorally Anchored Rating Scales 
(BARS) for each behavior. Finally, they reviewed and refined the curriculum during three 5-day 
expert panels that included ED physicians and nurses from 12 hospitals of various sizes.^^’^^ 

Training Objectives 

The overarching objective of MedTeams curriculum is to reduce medical errors through 

_ T'Xyf 

training interdisciplinary teamwork skills. MedTeams was developed on the premise that most 
errors result from breakdowns in systems-level defenses that occur over time. According to the 
MedTeams™ curriculum, each team member has a vested interest in maintaining patient safety 

TTVyf 

and is expected to take an assertive role in breaking the error chain. MedTeams defines a core 
team as a group of 3-10 (average = 6) medical personnel who work iriterdependently during a 
shift, and who have been trained to use specific teamwork behaviors to coordinate their clinical 
interactions. Each core team includes at least one physician and one nurse. A coordinating team 
manages several core teams, assigns new patients to the core teams and provides additional 

23,25^6 

resources as necessary. 


17 


Table 4. Summary of medical team training program content 


Evaluation Items 

Programs 


MedTeams™ 

MTM 

DOM 

Theoretical Basis 

> Based on the CRM* 
training program 
developed to train 

U.S. Army helicopter 
crews 

> Based on the U.S 

Air Force’s fighter 
pilot CRM training 
program 

> Based on CRM 
training from 
military and 
commercial 
aviation 

Needs Analysis 

> Analyzed the 
closed- case files 
from eight hospitals 

> Teamwork and 
communication as 
root cause of 
sentinel event 

> No known in- 
depth analysis 

Training 

Objectives 

> Reduce medical 
errors through 
interdisciplinary 
teamwork 

> Reduce medical 
errors by teaching 
human factors 
concepts to 
interdisciplinary 
teams of medical 
professionals 

> Change the 
military’s medical 
culture 

> Apply aviation 
safety practices to 
health care 

Training Content 
Knowledge 

Skills 

> Knowledge of the 
components of 
teamwork 

> Situational 
awareness 

> Maintaining team 
structure and climate 

> Problem-solving 
skills 

> Execution of plans 
and management of 
workload 

> Communication 
skills 

> Knowledge of the 
components of 
teamwork 

> Situational 
awareness 

Available resources 

> Policy/Regulations 

> Leadership 

> Identification of 
impaired 
performance 

> Workload 
performance 

> Communication 
skills 

> Knowledge of the 
components of 
teamwork 

> Situational 
awareness 

> Managing fatigue 

> Decisionmaking 

> Recognizing 
adverse situations 

> Cross-check and 
communication 


18 













Table 4. Summary of medical team training program content (cont.) 


Evaluation Items 

Programs 


MedTeams™ 

MTM 

DOM 

Skills (cont.) 

> Team 
performance 
improvement skills 
(e.g., teamwork 
review, situational 
learning, peer 
coaching) 

> Operating strategy 

> Recognition of 
obstacles to effective 
teamwork and 
communication 

> Performance 
feedback 

Instructors 

> Train-the-trainer 

> MedTeams"^ 
Certified instructors 

> Train-the-trainer 

> Crew Training 
International 
Instructors 

Instruction 

Strategies 

> 8 hours of 
classroom 
instruction 

> 1 hour of 
behavioral 
modeling, using 
videotaped 
vignettes 

> 1 hour of 
integration/synthe¬ 
sis 

> Didactic lectures 

> Seminar 
participation 

> Application 
questions 

> Behavioral 
modeling 

> Videotaped 
vignettes 

> Case study 
analyses 

>8 hours of classroom 
training. Including: 
lectures, demon¬ 
strations, case 
studies, and role 
plays 

Practice/Feedback 

> 4 hours of on-site 
practice 

> Routine team 
meetings 6 months 
after the training 

> Instructors’ 
observations and 
feedback 

> Homework 
assignments 

> Case studies 

> Development and 
Implementation of 
checklists and other 
tools 

Recurrency 

None 

None 

None 

Evaluation 

> Trainee reactions 

> Trainee reaction 

> Trainee reaction 

> Human Factors 

Attitude Survey 


* CRM = Crew Resource Management 


Training Content 

The MedTeams™ course consisted of a brief background on teams and an introduction to 
DoD Patient Safety initiatives. In addition, detailed information on six substantive modules was 
presented. Each module was structured around specific learning objectives. These objectives 
were likely derived from the results of the needs analysis that was described earlier but we could 
uncover no evidence to support this hypothesis. Finally, modules did not include any in-class 
assessments to ensure that participants had achieved the stated learning objectives. The six 
modules were as follows: 

1. Maintain Team Structure and Climate 

2. Plan and Problem Solve 


19 















3. Communicate with the Team 

4. Manage Workload 

5. Improve Team Skills 

6. Integration Unit 

The first module (“Maintain Team Structure and Climate”) included information about the 
composition of core teams, team leader and team member roles, team structure, and team 
climate. It also presented a technique for managing conflict: the DESC Script, which encourages 
(describing the situation, expressing concerns about the action, .suggesting alternatives, stating 
consequences, and obtaining consensus. 

The “Plan and Problem Solve” module taught skills such as planning, using shared mental 
models, cross-monitoring, using assertion and advocacy, and using the two-challenge rule. 
Planning consisted of both long-term and situational planning, such as responding to 
emergencies. Shared mental models ensure that team members have the same understanding 
about the situation and/or problem. The remaining discussion focused on encouraging team 
members to voice their concerns in order to prevent errors. 

“Communication” was the theme for the third module. This module taught information about 
situational awareness and the standards of effective communication (e.g., clear, timely, complete, 
and verified). Information transfer skills that encourage clarification of information, such as 
check-backs, call-outs, and hand-offs, are also part of the communication module. 

“Managing Workload” was the fourth section. This module focused on workload information 
and skills for managing workload, such as resource management, prioritization, delegation, and 
task assistance. 

The “Improving Team Skills” module consisted of information about performance goals and 
feedback, including characteristics of effective feedback. Performance improvement skills, such 
as teamwork reviews conducted at the end of a clinical event or near the end of a shift, 
situational teaching, and peer coaching, were explained. Situational learning involves 
encouraging questions, while peer coaching consists of monitoring team members’ performance 
and instructing team members. 

Finally, the last unit was the “Integration Unit.” The purpose of this module was to discuss 
implementation issues such as staff expectations of implementation, team implementation plans, 
and sustainment issues, such as the strengths and weaknesses of teamwork systems and action 
plans for managing obstacles to implementation. In addition, the curriculum contained a 
teamwork simulation, in which the class was divided into a simulation team and an observation 
team. The instructor selected the scenario from a Simulation Matrix, the simulation team enacted 
the scenario using teamwork skills, and the observation team then rated the team on its teamwork 
actions. Time permitting, students then changed teams and enacted another scenario. This 
module was designed to be an open discussion among the students; however, the simulation was 
cut short or not done in many of the classes due to time constraints. 


20 


Instructor Selection, Training, and Preparation 

Description. As mentioned previously, MedTeams™ uses a train-the-trainer approach to 
implement the training. Individuals designated by their facility receive comprehensive training. 
DRC requests that designated instructors be: 

• Viewed as advocates of teamwork 

• Members of a physician/nurse/technician training team 

• Viewed as leaders among their peers and service administrators 

• In positions that allow flexibility in scheduling 

At the end of the train-the trainer course, these individuals are certified as MedTeams™ 
instructors. Certification requires instructors to complete MedTeams training, complete the 
Instructor Certification Course, complete the practice teaching and coaching prerequisites, and 

‘TTVyf 

pass a written exam with a score of no less than 80 percent. MedTeams instructors are then 
responsible for implementing MedTeams training at their health care organization. 

Findings. Observations of the train-the-trainer course in Boston and a review of the L&D 
instructor manual indicated that MedTeams uses a variety of control mechanisms to ensure 
instructor quality. These include standards for instructor selection, an instructor certification 
course, opportunities to practice and receive feedback on teaching and coaching, and an 
assessment of team and course knowledge through a written test. We conclude from reviewing 
this information that this train-the-trainer strategy should produce knowledgeable instructors who 
provide high quality, reliable training at their facilities. 

XTVyf 

To cross-validate our finding, we interviewed 14 MedTeams -certified instructors who 
conducted training at the naval medical center we visited. As part of these interviews, we asked 
these individuals how they were selected and how they were prepared to teach the OR course. 
Findings from these interviews are presented in the following paragraphs. 

First, regarding instructor selection, interviewees provided a variety of reasons as to why 
they were selected to teach the MedTeams™ curriculum for the OR. For example, interviewees 
reported that department heads were required to teach at least one session of the course, but a 
few department heads delegated their responsibility to other people. Some instructors thought 
that senior physicians were chosen to teach the course in order to encourage attendees to focus 
on the course. A few thought that they were chosen at random; others thought that they were 
selected due to previous training experiences. Two individuals felt that they were chosen because 
they are the continuous improvement (Cl) representatives for their de^rtments. Two instructors 
were involved with the development of the OR version of MedTeams™; thus, they were willing 
to participate. 

Regarding preparation, instructors spent a variety of time periods preparing to teach the OR 
course. Most spent between 2 and 6 hours, with several spending 8 hours or more in planning. 
Most instructors spent between 2 and 4 hours as a group preparing to conduct the class, and then 
spent additional time reviewing the curriculum on their own. 

Besides the MedTeams™ instructor certification course, there was no additional training or 
preparation required to teach the course. However, even with the extensive activities included in 
the certification course, most instructors did not feel adequately prepared to conduct 


21 


‘TXyf 

MedTeams training. Several mentioned that they were not aware that they would become 
instructors for the course until the end of the train-the-trainer session, even though at the onset of 
the train-the-trainer course they were notified that they would ultimately be teaching 
MedTeams . Most felt that they needed more time to become familiar with the materials; 
several mentioned that they did not receive the materials until the end of the week before they 
were supposed to teach the course. In contrast, another group of instructors felt well-prepared, 

TTA A 

but they also had familiarity with MedTeams prior to teaching the course. 

Instructional Strategies 

__ *T*\yf 

Description. MedTeams purports to employ a variety of training methods that address two 
of the three recommended phases of CRM training: Awareness and Practice-and-Feedback. The 
Continual Reinforcement phase is not directly addressed, although it could be argued that this 
occurs through the on-going practice of MedTeams training and the implementation of several 
sustainment strategies. 

Regarding awareness, this phase includes 8 hours of classroom instruction, 1 hour of 
behavioral modeling using videotaped vignettes, and 1 hour of integration/synthesis. Once the 
classroom training portion of the course is complete, each team member participates in a 4-hour 
practicum that involves practicing teamwork behaviors and receiving feedback from a trained 
instructor (Practice and Feedback Phase). Coaching, mentoring, and review sessions are also 
provided during regular work shifts. The post-classroom component of training lasts for 
approximately 6 months. 

Findings. We observed portions of 11 classes during our visit to the naval medical center 
during summer 2003. Classes varied from 3 to 7 hours. Classes varied primarily as a function of 
the instructor and how that individual chose to conduct the course. For example, some instructors 
simply read the slides and did not add much detail or explanation. Other instructors used a more 
interactive style, engaging the attendees in discussions, obtaining examples from their 
specialties, and even calling on individuals when necessary. 

Most instructors utilized the vignettes to demonstrate key points during training; a few 
instructors even customized or wrote new vignettes tailored to their specialty (e.g. 
ophthalmology). Most instructors showed the videos. However, few, if any, instructors actually 
conducted the practice exercises. The primary reason for skipping these activities was the lack of 
time. 

In addition to our observations, we asked students to indicate the extent to which they agreed 
with the statement, “The training was well organized.” Somewhat in contrast to what we 
observed, 96 percent of the students agreed or strongly agreed with this statement (see Table 4). 
The vast majority of students also reported that the training content was appropriate for their 
department. 

In summary, it should be noted that we were only able to view the in-class portion of 
MedTeams training. Observations and other data were not collected on any posttraining 
implementation strategies. Regarding the classroom component, our observations and instructor 
interviews suggest that there was wide variation in how instructors implemented MedTeams™ 
training. Overall, classes varied from 3 to 7 hours. Even though instructors were trained and 
certified, how the course was implemented and conducted was left to the discretion of the 
instructors and the quality of instruction varied considerably. The instructional strategies that we 
observed (lectures, case studies, and video demonstrations), when they were implemented 


22 


properly, seemed to be effective at achieving the desired objectives. However, because one of the 
core objectives of MedTeams™ is to enhance the team skills of the participants, we would have 
liked to see more instructors implement the role-play exercises. Nonetheless, student reactions to 
the organization of the course and its appropriateness were extremely positive. 

Training Effectiveness 

Description. The classroom-based phase of MedTeams™ primarily relies upon the 
collection of trainee reactions (Kirkpatrick’s Level I data) to determine training effectiveness. 

The Emergency Team Coordination Course® Evaluation Form (© 1997 Dynamics Research 
Corporation) is used for this purpose. For the OR course, trainees were asked to rate the extent to 
which MedTeams training achieved its purpose and goals, met its stated objectives, and was 
well organized. In addition, participants were asked to assess the quality of the instruction. One 
item addressed the expertise of the instructor, and the other addressed the appropriateness of the 
instructional strategies for achieving the desired objectives. 

TTi a 

Findings. To assess MedTeams effectiveness, we collected additional, independent data 
on both trainee and instructor reactions to the course. Utility reaction data were collected from 
participants and, as mentioned, interviews were conducted with instructors. In addition, we 
reviewed the existing empirical literature that has been reported regarding Kirkpatrick’s Level II 
(Knowledge), Level III (Behavior), and Level IV (Results). Specifically, we reviewed a quasi- 

_ 'T'X/f 

experiment that examined the effectiveness of MedTeams in the ED that was conducted by 

Morey and colleagues. Below, we describe our findings from these activities. 

Level I data. Participants in the OR course responded favorably to the training. Overall, 
participants liked the content of the course and felt that it was useful. The course met their 
expectations, in part, due to the content and issues, with respect to departmental problems and 
communication problems that were raised. More specifically, attendees felt the content was well- 
organized (96 percent agreement; see Table 5) and appropriate for their department (92 percent 
agreement). Almost all attendees felt confident that they understood the material, could perform 
teamwork tasks, and could apply the course material to their jobs. Though these ratings also were 
high, there was slightly less agreement that the training was an effective use of time (77 percent) 
and that the training prepared attendees to work effectively in their jobs (78 percent). 


23 


Table 5. Posttraining opinions about MedTeams^^ 



Total Responses (n-21d) 

Item 

Mean 

Std. 

Dev. 

Percentage 

Agreement 

Percent¬ 

age 

Neutral 

Percentage 

Disagree¬ 

ment 

The training was well-organized. 

4.4 

.68 

96% 

2% 

2% 

1 am confident that 1 can perform 
the tasks that were trained. 

4.4 

.58 

96% 

3% 

1% 

1 am confident that 1 understood 
the training content. 

4.5 

.60 

96% 

3% 

1% 

1 am confident that 1 can use the 
knowledge that 1 learned on the 
job. 

4.4 

.69 

94% 

5% 

1% 

The training content was 
appropriate for my department. 

4.3 

.72 

92% 

6% 

2% 

Training will help my department 
improve patient safety. 

4.1 

.81 

83% 

14% 

3% 

As a result of this training, 1 feel 
more confident about my ability to 
work effectively In a team. 

4.1 

.83 

80% 

17% 

3% 

Training prepared me to work 
effectively in my job. 

4.0 

.82 

78% 

18% 

4% 

Training was an effective use of 
my time. 

3.9 

.95 

77% 

14% 

9% 


When asked about any changes that could be made to the course, a little less than half of 
attendees felt that nothing should be changed. Those who felt that changes should be made 
wanted more interaction with others, including group activities and role-plays, and more videos, 
vignettes, and case scenarios. Several participants suggested shortening the course, and a few felt 
that the course was redundant in parts and could be condensed. 

Almost all attendees would recommend this training course to other people at work. The 
main reason for recommending the course is the need for other co-workers to learn about 
teamwork and improve their teamwork skills, particularly communication. 

Likewise, instructors were pleased with the course. They felt that the sequence and flow of 
the course were appropriate. The training aids (e.g., videos, vignettes) were also identified as 
being useful. A few instructors noted that Module 1 (“Maintain Team Structure and Climate”) 
appeared to be the most important and that the videos in the beginning helped to gain trainees’ 


24 



















attention. However, a few instructors felt that there were almost too many videos and that it 
might be better to include fewer. Suggestions for other training aids included more interactive 
handouts that encourage participation and vignettes with OR-relevant examples. Better audio¬ 
visual coordination was another suggestion. 

Instructors felt that the communication module and the fmstration exercise were two of the 
best features of the course. The frustration exercise (i.e., having the group voice their fhistrations 
with the OR) proved to be very useful. Participant fimstrations were written down on a flip chart, 
and several instructors linked the fhistrations back to specific components of MedTeams™ 
towards the end of the course. This activity added validity to the MedTeams™ training by 
showing how it can help alleviate or reduce some of the attendees’ frustrations. Other key 
features that were cited by instructors included: 

• Pre-packaged materials (e.g., slides and student’s manual) 

• Videos 

• Basic nature of training (i.e., easy to understand) 

• Training crosses all skill levels/allows for face time with all groups involved in the 
OR/gets surgeons involved 

• Patient-safety focus of training 

• Identification of solutions to common problems 

Instructors felt that less time could be spent on self-explanatory information. According to 
instructors, specific components of the course that could use improvement include Modules 4 
and 6 (“Manage Workload” and “Integration”). Module 4 is dense and difficult for many 
participants to understand; several felt that Module 4 (“Workload”) should be revised and 
shortened. In particular, the concept of “task assistance” was particularly difficult for some 
trainees to grasp. Module 6 does not do a good job of wrapping up the course, according to 
several instructors. The simulations (i.e., role-plays of teamwork skills) included in Module 6 
encourage lengthy discussions instead of reviewing the key points of the course. 

Almost all instructors noted that the examples in the course should be made more relevant to 
an OR. A few instructors recommended improving the audio-visual integration; one suggested 
including markers in the text to cue instructors for upcoming videos and vignettes. Other 
recommendations from the instructors included the following: 

• Include the fhistrations exercise in the curriculum 

• Re-arrange the curriculum to put the most important information up front 

• Consider the OR environment (some terms are too “touchy-feely” or “hokey”) and 
address people’s roles 

• Improve organization and planning for the course (e.g., the lack of preparation time) 

• Place greater emphasis on the following items: 

° Integration (i.e.. How will the training be implemented? How will it work?) 

° Communication 

° Improvement of team skills 


25 


° OR-based videos and practical exercises 

° Involvement of the surgeons (e.g., focusing on the patient and working with the 

team) 

Finally, interviewers asked the instructors what the likelihood is that the training will be 
successful in their departments. Several believed that there would be a 40-50 percent chance of 
success due to anticipated difficulties with implementation. One of the major concerns voiced 
was that individuals in the OR do not have enough time or staff to keep the training going and 
ensure its successful implementation. To address this issue, one instructor suggested moving the 
responsibility for the training course to the Staff, Education, and Training Group (SEAT). 
Another concern was that many of the surgeons had not bought into the training and felt that it 
was a “waste of their time;” thus, they might not be willing to implement it. A third concern was 
the length of the training and the need for refresher training. Additionally, several instructors felt 
that the DoD should put more resources into the training (e.g., organization and preparation). In 
contrast, a few instructors felt that there was high likelihood that the training would work, but it 
would depend on training and sustainment. According to many instructors, as long as leadership 
supports the training, it will be implemented, and everyone new who comes in will get the 
training. 

Level II, III, and IVdata. There has been one major evaluation of the*MedTeams approach 

in emergency departments and another is underway in labor and delivery units. The ED 
study^^ involved a multisite, single-crossover, quasi-experimental design. In this study, nine EDs 
(six in the experimental group and three in the control group) were observed during a 14-month 
interval that encompassed pretraining baseline measures, the training intervention proper, and 
posttraining evaluations. A suite of 17 process and performance measures was collected. To 
ensure rating accuracy, all observation-based measures were collected by trained raters, and 
measures of interrater agreement were periodically conducted to ensure that the raters remained 
calibrated. Finally, because data were clustered. Generalized Estimating Equations (GEE) were 
used to test the effect of the hospital-level intervention using case-level data. The results 
suggested that, in contrast to the control group, the trained groups showed significant gains in 
teamwork-related knowledge, skills, and attitudes; that the intervention did not increase self- 
reported task workload; and that the error rate decreased sharply. 

The second study is ongoing in L&D units in civilian and military hospitals.^^ Unlike the 
previous study, in which the EDs chose to participate in either the experimental or control 
conditions, this study was designed as a randomized clinical trial. Based on an a priori power 
analysis, 24 hospital L&D units were randomly assigned to participate in either the experimental 
or control conditions (up to 12 per condition). Many other aspects of the L&D study mirror that 
of Simon and colleagues^"^ with multiple performance measures that focus on patient outcomes, 
team process, and staff and patient satisfaction. However, unlike the ED investigation, 
performance ratings on the team skills taught in MedTeams will not be collected. Rather proxy 
measures, such as the time it takes for new patients to be processed through hospital admissions 
and the time interval between deciding to do a C-section and initial incision, will be recorded. 
Data collection for the L&D study was expected to be completed in the spring of 2004, however, 
the findings have not yet been published. 

Strengths and Weaknesses 

Strengths. Our review of the literature, observations of the classroom phase, instructor 
interviews, and posttraining assessment suggest that MedTeams™ has a number of desirable 


26 


qualities. First, the original courseware for the ED was based on a thorough up-ffont needs 
analysis. This analysis resulted in the five core dimensions on which MedTeams™ training is 
based. Second, MedTeams employs a very practical system for implementing the training. 
Designated staff members from a specific facility are trained and certified as MedTeams™ 
instructors, and then these staff members conduct MedTeams™ training at their facilities (i.e., a 
train-the-trainer strategy). Moreover, our review of the “L&D Team Coordination Course” for 
instructors suggests that instructor training is comprehensive and thorough*. Third, our 
independent collection of posttraining reactions suggests that participants had positive reactions 
to MedTeams™ training. Participants indicated that the training was well organized, and they 
felt that they could use many of the strategies that were discussed during training upon returning 
to their jobs. Finally, the MedTeams developers have made the most extensive efforts to 
collect Level II and III data to demonstrate the effectiveness of MedTeams™.Although the 
initial investigation suffered several design flaws, which make the results somewhat tentative, it 
was one of the few efforts in the academic literature to link team process to health outcomes. 
Hopefully, the L&D study will shed additional light on these relationships and the utility of 
MedTeams™. 

_ TTVyf _ 

Weaknesses. Nevertheless, MedTeams does have its limitations. First, although a 
comprehensive needs analysis was performed to develop the ED curriculum, no subsequent in- 
depth needs analyses were conducted to develop the L&D and OR courses. Essentially, subject 
matter experts reviewed the training materials and customized the case studies and other 
examples when appropriate. We question this approach, because a panel of leading experts in the 
field that was convened by AIR in January 2002 suggested that team knowledge, skills, and 
attitudes are likely to vary by medical specialty as well as other factors (see Appendix D). 
Second, we were surprised by the variation in how the classroom phase was administered by 
trained, certified MedTeams instructors. Classes ranged from 3 to 7 hours and the quality of 

TTVyf 

this instruction varied greatly. Despite the MedTeams developers’ best efforts to ensure 
consistency of instruction, consistency was lacking. Also, while one of the main objectives of 
MedTeams™ is to develop team skills, much of the classroom instruction focused on mastering 
declarative and some procedural knowledge. There was substantially less time devoted to skills 
practice. Third, MedTeams did not employ a cultural assessment/evaluation component prior 
to implementing the training. As a result, it is entirely possible that MedTeams is effective 
only in hospitals that have already made a commitment to teamwork, secured upper-level 
management support, established an open, nonpunitive atmosphere that embraces errors as an 
opportunity for learning, and recognized the need for change. It is interesting to note that results 
from our pretraining assessment suggest that this particular naval medical center has a culture 
that supports teamwork. Fourth, trainee reactions to MedTeams™ were positive despite the 
tremendous variability in instruction. However, and quite in contrast to the positive reactions, 
several of the instructors we interviewed said that there was only a 40-50 percent chance of this 
training being successful when implemented. Finally, one of the limitations of MedTeams is 
the delay in the implementation of the actual strategies. All departmental staff must receive the 
classroom phase before implementation. Although this appears to be a reasonable approach, 
especially when training a large number of people, there can be considerable delay between 
classroom training and implementation. For example, we visited the naval medical center again 
in early July 2003. At that time, OR staff training had not been completed. Such a delay could 
result in a decay of important knowledge and strategies that were developed during training. 


27 


Summary 

Trx n 

In conclusion, the MedTeams course was well received. The content covered the basics of 
teamwork and, for the most part, students and instructors felt that it delivered a good message: 
teamwork is important. The course met students’ expectations, and most would recommend the 
class to coworkers. The frustration exercise was identified as the most popular activity in the 
class. Both students and instructors liked the interaction between physicians, nurses, and medical 
technicians that resulted from the exercise. They felt that it was a good way to raise awareness 
about issues and concerns from different health care professionals’ perspectives. It also tied in 
nicely with the modules of the course, as many of the concerns were addressed in the modules. 
However, instructor variability plagued the actual conduct of the training. Moreover, many 
attendees were frustrated by the length of the course as well as the over reliance on lectures as 
the primary instructional strategy. Many would prefer to break the lecture portion up with more 
videos, case scenarios, and group discussions. In addition, many wanted more examples that 
were directly relevant to the OR. 


28 


Chapter 3. Case Study 2—Medical Team Management* 

Introduction 

The primary purpose of Medical Team Management (MTM) is to reduce medical errors 
through interdisciplinary team training. MTM was modeled on the U.S. Air Force’s CRM 
training program for fighter pilots and was developed after poor teamwork was identified as the 
root cause of a medical event that led to a catastrophic patient outcome. 

The MTM training program has two major components: a 3-day train-the-trainer course and 
a medical treatment facility course. Upon completing the train-the-trainer course, graduates 
return to their respective medical facilities to train the remaining staff in teamwork principles.^^ 
The MTM curriculum includes an introduction to the program, overviews of key patient safety 
and CRM issues, and specific modules for seven foundational elements: leadership, workload 
performance, policy and regulations, situational awareness, available resources, communication 
and operating strategy. In addition the curriculum includes obstacles to effective teamwork, and 
tools (behaviors) for improved teamwork and communication. Case studies, vignettes, and tools 
(e.g., the “two attempt” rule) are interspersed throughout the curriculum to reinforce the 
importance of effective teamwork. 

Case Study Approach 

We were able to access several sources of information to assess MTM. Specifically, we 
reviewed the course materials for the MTM train-the-trainer course, the MTM Handbook, and 
the MTM implementation guidelines. In addition to our document review, we observed the 3-day 
train-the-trainer course, collected pretraining data on student experiences and expectations, 
collected posttraining data on student reactions to MTM, and interviewed several current and 
former participants. Each of these activities is described in more detail below. 

Document and Literature Review 

As part of a larger state-of-the-art literature review, AIR conducted a document review on 
medical team training and specific medical team training programs, including MTM. Unlike 
MedTeams™, there were no published articles, conference proceedings, or technical reports that 
document the effectiveness of MTM. However, we carefully reviewed the MTM train-the-trainer 
toolkit, the MTM Handbook, and the MTM implementation guidelines, which were provided to 
us prior to attending the train-the-trainer course. 

Course Observations 

AIR staff members observed the MTM train-the-trainer course, which took place at a U.S. 

Air Force base during summer 2003. As described in the MTM curriculum, the train-the-trainer 
course lasted 3 full days. The first day consisted of an introduction to MTM, along with modules 
on the foundational elements. The second day focused on the obstacles to effective teamwork. 


* Note: Appendixes cited in this report are provided electronically at 
http://www.ahra.aov/Qual/teamtrain/index.html#app . 


29 




MTM tools, case studies, videotaped vignettes, and implementation guidance. The final day 
consisted of student-led case studies, a posttraining wrap-up, and student-led feedback. 

Pre- and Posttraining Assessment Tools 

In addition to observing the MTM train-the-trainer course, AIR staff administered the pre- 
and posttraining assessment tools to MTM participants. The pretraining tool was designed to 
assess experiences prior to training and to gauge attitudes and beliefs about teamwork. The 
posttraining tool measured participant utility reactions to MTM.^ The attendees represented a 
variety of medical specialties including anesthesia, cardio-thoracic medicine, dentistry, 

OB/GYN, internal medicine, physical therapy, emergency medicine, and pharmacy. 
Approximately one-third of them were currently involved in some form of patient safety or 
quality assurance programs at their respective medical facilities. Of the 26 participants surveyed, 
3 were physicians, 12 were nurses, and 11 represented other groups such as medical technicians, 
physical therapists, pharmacists, and dental technicians. 

Participant Interviews 

During our visit, we also interviewed 10 individuals, two of whom had attended a previous 
train-the-trainer course. Participants were asked about how they were selected to teach MTM, 
their training background, and their perceptions of the flow of the course and training aids which, 
were used. In addition, they were asked to name the three best features of the course and three 
that could use improvement, as well as any major obstacles to implementation. Finally, they 
were asked how they felt their coworkers might respond to the MTM course, and when they 
expected the training to be implemented at their respective medical facilities (see Appendix C). 

Below we present the results of this case study. These results are organized around the same 
set of variables that we used to draw conclusions about MedTeams . For each variable, we 
provide descriptive data from our literature review and then findings from our observations, data 
collections, and interviews when appropriate. Table 4 (p. 18-19) provides a summary of this 
information for each of the three training programs that we reviewed. 

Results 

Theoretical Basis 

The core objectives, curriculum, and instructional strategies for MTM were derived from the 
U.S. Air Force’s aviation CRM training course. Like MedTeams , the MTM course developers 
argued that medicine and aviation share a number of similarities, thereby making CRM training 
for pilots portable to health care. However, unlike MedTeams , which has been tailored to 
specific medical specialties (e.g., ED, OR, L&D), MTM was designed to be a generic course that 
can apply equally well in a variety of medical specialty and support areas with little or no 
modification. 

Analysis of Training Needs 

MTM was developed after an adverse event resulted in a newborn developing severe 
neurological problems.^^ Although not part of a formal needs assessment, the need for 


^ We administered posttraining tools after the second day of training. Even though this was a train-the-trainer class 
and participants were future MTM instructors, the first 2 days of the course were devoted to demonstrating MTM 
training. 


30 



communication and team training was validated by a subsequent review of 60 closed cases. The 
reviewers identified poor communication as the primary cause (74 percent) of otherwise 
preventable adverse events, providing further support for MTM. Drill-down analyses suggested 
that these 60 cases resulted in 92 separate errors. Furthermore, in cases where communication 
was an issue, there was an average of two errors per case.^ 

Training Objectives 

The primary purpose of MTM is to reduce medical errors by teaching human factors 
concepts to interdisciplinary teams of medical professionals.A secondary purpose is to 
change the military’s medical culture. Traditionally, the culture has focused on individual 
performance and, in doing so, has created obstacles to communication. MTM was specifically 
developed to foster a culture that values team performance and encourages effective 
communication across medical specialty areas and throughout the chain of command. 

Training Content 

The MTM training program covers seven foundational elements: 

1. Leadership 

2. Workload Performance 

3. Policy and Regulations 

4. Available Resources 

5. Situational Awareness 

6. Communication 

7. Operating Strategy 

In addition, obstacles to effective communication and teamwork and tools, or desired 
behaviors, are critical elements taught in MTM . All are designed to improve participants’ 
knowledge and skills in these core areas. 

The “Leadership” module highlights the differences between leadership and authority, and 
provides techniques for regulating information flow, directing team activities, motivating team 
members, and making effective team decisions. The “Workload Performance” module highlights 
the curvilinear effect of workload on performance—that performance worsens when workload is 
either too high or too low. It also provides techniques for identifying and dealing with high levels 
of workload, distraction, stress, and fatigue. The “Policy and Regulations” module focuses on the 
importance of following guidelines for maintaining patient safety, the various reasons for 
departing from established guidelines, and the consequences of doing so. The “Available 
Resources” module discusses using all assets, such as team members, equipment, skills, and all 
information available to the medical team. It describes both internal and external resources and 
emphasizes that thorough and effective use of these resources maximizes safety. The “Situational 
Awareness” (SA) module defines the concept of situational awareness, provides cues to 
determine if team members have lost SA, and suggests techniques for maintaining SA. The 
“Communication” module identifies the various forms of verbal and nonverbal communication. 

It also identifies principles for effective communication, obstacles to effective communication, 
and real-life examples of communication problems between health care providers. The 
“Operating Strategy” module provides recommendations for integrating the various teamwork 


31 


issues. It also highlights the role of shared mental models, which develop as team members 
interact over time and shows how shared mental models can improve team performance. The 
“Obstacles to Effective Teamwork” module identifies a number of factors—such as excessive 
professional courtesy, the halo effect, and hidden agendas—that cause breakdowns in team 
communication and set the stage for medical errors. The final module teaches tools, or desired 
behaviors such as the “FM SAFE checklist,” “assertive statement,” and “two attempt” rule for 

combating obstacles and improving teamwork. 

1 

Instructor Selection, Training and Preparation 

Description. Our review of MTM documents found that the U. S. Air Force requests that 
candidate MTM instmctor have at least 5 years of clinical experience in their specialty areas. In 
addition, instructors should have at least 1 year of retainability in the armed forces, must be a 
competent speaker, and are expected to have previous experience delivering training. 

Findings. Although MTM has specific requirements for MTM instructors, we could not 
identify any formal procedure for selecting potential MTM trainers. Our interviews with the 
MTM participants suggested that most were either recommended by a senior officer, or had 
volunteered because of their personal interest in maintaining patient safety. The participants 
came from a variety of medical specialties (from surgeons to medical technicians), and included 
enlisted to mid-level officers (from E-5s to 0-6s). This mix was not unexpected because MTM 
was designed to be an interdisciplinary training course, which encourages open and effective 
communication, regardless of rank and content domain. 

Instructional Strategies 

Description. MTM employs a variety of training methods. These include computer-based 
instruction (which has temporarily been suspended), didactic lectures, seminar participation, 

o ^ ^ /I 

application questions, behavioral modeling, and case studies. ’ ’ The trainees are also required 
to complete a variety of homework assignments. One involved observing one’s own team to 
identify obstacles that hinder effective team performance. Another requires the trainees to 
practice the tools that they have learned in the workplace. The trainees then identify the lessons 
that they have learned and discuss them at subsequent training sessions.^^ 

Findings. During the train-the-trainer course, we observed didactic lectures, seminar 
participation, application questions, and case study analyses. The case studies were a significant 
portion of the class, as the instructors went over the technique on the second day and assigned 
students to develop their own case study for the next day. On the third day, the participants 
discussed their case studies in small groups. The participants generally seemed to enjoy this 
exercise, and many recommended including more case studies in future MTM train-the-trainer 
courses (see Table 6). 

A number of techniques for sustaining and reinforcing the human factors concepts discussed 
were also included in MTM training. For example, a module on the third day was devoted to 
sustainment issues, especially in unanticipated situations. Topics included long-term planning, 
briefings, and continuous monitoring of operations. 

In summary it should be noted that we were only able to view the in-class portion of the 3- 
day train-the-trainer course for MTM. Therefore, unlike MedTeams™ and DOM, trainees were 
future MTM instructors not participants. Our observations suggested that the first day of MTM 
training was an orientation day; essentially instructors received MTM training. Therefore, we 


32 


believe that the results we collected on our posttraining measure are representative. With that in 
mind, we now turn to these findings. 

Training Effectiveness 

Description. The MTM instructors administered an evaluation form at the end of the training 
course. The form included demographic information such as participants’ position, employment 
status (i.e., active duty or civilian), and experience level. Participants were then asked to rate the 
various pieces of the course and the trainer toolkit, using a four-point scale, with anchors ranging 
from “not helpful at all” (1) to “very helpful ” (4). They also rated the effectiveness of the 
instructors and gave an overall course rating, using a three-point scale, with anchors ranging 
from “marginally effective” (1) to “very effective” (3). In addition, they were asked to rate their 
self-efficacy at teaching and applying MTM principles in their respective medical facilities using 
a four-point scale, with anchors ranging from “very uncomfortable” (1) to “very comfortable” 

(4). Finally, they were asked to identify the one thing that was most helpful and the one thing 
that was least helpful in preparing for the course. 

Findings. Level I data. Overall, the MTM course was well received. Participant reactions 
indicated that it was well organized and contained appropriate content. Many attendees exuded 
confidence from the training course; they felt confident that they understood the material and 
could perform the teamwork tasks and apply them to their jobs (see Table 6). The course met 
their expectations because it clearly presented teamwork information and the goals of the 
program. In addition, it gave attendees skills to use to help improve teamwork in their 
departments. 

Attendees were also pleased with the sequence of the course. They were particularly pleased 
that the material was presented and then followed by examples and/or videos to support it. The 
videos, vignettes, and case studies helped reinforce the material for the attendees. However, 
many of the videos and vignettes were examples of poor teamwork skills; attendees noted that 
examples of excellent teamwork skills should also be included. In fact, one attendee suggested 
having each facility participating in the training submit a teamwork success story ahead of time 
that could be discussed at the course in order to emphasize the results of effective teamwork. 
However, it could be difficult for attendees to pinpoint such examples. 

Attendees felt that the three major strengths or best features of the course include the 
material, particularly the communication section, the videos, vignettes, and case studies, and the 
speakers. They also liked the audience participation and interaction both within the smaller 
groups and with the instructors. Other features that were named include the templates in the 
manual and the online portion of the course, where students are able to go through some of the 
material at their own pace before they attend the course. Unfortunately, the online training aid is 
not currently available. Several attendees would like to have the online section available as both 
a preview of the information for those who will be attending the class and also as refresher 
training for those who have already taken the course. 


33 


Table 6. Posttraining opinions about MTM 



Total (n=26) 

Item 

Mean 

Std. 

Dev. 

Agreement 

Neutral 

Disagreement 

The training was well organized. 

4.6 

.49 

100% 

- 

- 

1 am confident that 1 understood 
the training content. 

4.5 

.59 

96% 

4% 

- 

1 am confident that 1 can perform 
the tasks that were trained. 

4.2 

.72 

92% 

4% 

4% 

1 am confident that 1 can use the 
knowledge that 1 learned on the 
job. 

4.3 

.80 

88% 

8% 

4% 

As a result of this training, 1 feel 
more confident about my ability to 
work effectively In a team. 

4.2 

.65 

88% 

12% 


The training content was 
appropriate for my department. 

4.2 

.75 

88% 

8% 

4% 

Training will help my department 
Improve patient safety. 

4.1 

.60 

88% 

12% 

- 

Training was an effective use of 
my time. 

4.1 

.70 

88% 

8% 

4% 

Training prepared me to work 
effectively In my job. 

4.0 

.71 

76% 

24% 

- 


Nearly all attendees would recommend the course to their fellow co-workers, because they 
felt that everyone could benefit from teamwork training. In addition, the course emphasizes 
critical skills that are vital for safe and effective patient care. However, two-thirds of attendees 
would change the course if they were to re-design it. The most popular suggestions include 
adding more case studies, scenarios, and videos as examples of both poor and effective 
teamwork and deleting the aviation CRM videos. Several attendees felt the aviation videos were 
difficult to understand and felt that explaining how CRM in aviation relates to medical team 
training was sufficient. A few attendees felt that the course was too long and there was too much 
time devoted to lecture. Attendees also felt they needed more answers to difficult questions and 
resistance they may face with leadership upon their return from the course and attempts at 
implementation. A final suggestion made was to change the case study template to ensure that it 
matches the order in which the foundational elements are presented in the manual. 

The two biggest obstacles attendees named were getting buy-in from leadership and 
scheduling the training. The difficulty in convincing leadership stems from the challenge of 
showing them tangible values and benefits to the training, such as a decrease in the number of 


34 



















medical errors made or a large amount of money saved due to the training. With regards to 
scheduling, it is difficult to make arrangements for numerous people from the same department 
to attend an all-day course; it limits patients’ access to health care. Also, deployments and 
assignment changes make it difficult to get everyone trained. A few people thought that the 
training should be mandatory to have a real effect on patient care. 

Level II, III, and IV data. Unlike MedTeams , we could not identify any additional data in 
the published literature on MTM describing its effectiveness beyond trainee reactions. 

Strengths and Weaknesses 

Strengths. To summarize, MTM offers a number of advantages. First, it uses a series of 
active learning techniques—including didactic lectures, behavioral modeling, and case studies— 
to develop trainees’ teamwork-related knowledge, skills, and attitudes. Second, it leverages 
known principles from human-factors research. For example, MTM training (a) explicitly 
distinguishes between destructive and constructive conflict resolution, (b) recognizes that the 
workload-performance relationship is curvilinear, and (c) distinguishes between authority (which 
is based on rank) and leadership (which is based on skills). Third, MTM training is 
interdisciplinary in nature, thereby teaching physicians, nurses, technicians and other key 
constituencies to work together. Finally, it provides a reference list that allows participants to 
continue refining their teamwork skills after they have completed the training. 

Weaknesses. Nevertheless, like the other two programs, MTM has disadvantages. First, far 
more of the training time is devoted to providing factual information than to practicing actual 
skills with instructor feedback; the skills practice that is provided primarily involves low-fidelity 
techniques such as case studies. Second, although MTM provides trainees with a variety of 
“tools” to reinforce and sustain their teamwork skills, many of these aids are not tools in the 
strictest sense of the word. More often than not, the MTM materials consist of best practices or 
procedures (e.g., briefings, cross-checks), but not tangible tools (e.g., checklists, quick reference 
cards, etc.) that trainees can physically take with them. Third, even though MTM is based on the 
“train-the-trainer” paradigm, it does not appear to include mechanisms for preventing 
performance degradation among trainers. Finally, there does not appear to be a formal recurrency 
module for ensuring the maintenance of trained knowledge and skills. 

Summary 

In conclusion, attendees enjoyed the MTM course. It made them more aware of teamwork 
issues and taught them valuable skills. The most common suggestion for improvement that 
attendees made was to include more vignettes and case studies and even cut down the lecture if 
necessary. Finally, participants also highlighted the importance of U.S. Air Force leadership 
support to ensure that MTM is successful. 


35 




f ^ 




irvjis. 


’« 


>;.,' -1 



»■ y 




.T ■ ‘ i ■ ■ ■ I'« • f' ?<<*•■" 










»■' 1 " I 'ij •v»*< I'f'' 

A 'u-tI ■ '^ 


-. S' ' 


■«•• J*.- •• (', 









i 


* '*'» 




m . * 

.. v.wkr ■ - B 

• r* ' -*Kr * ' ' ^ •A, ' ♦*A -4 ► * ^ 




■P*LV. •'•■*■ ■ A'^p-r.s' . •■:. 

•f-. ^ 

3 iiiii}i|^^ lir»« 









if T l_ 1 - • ■ ■■ 

■‘ll(!.:v!*6wWfS!ir'«J,’K(^l^ 

r« ■ ^ ■■ •■' •' 

:} V iv ; /. . 






r > 




ii J u** #1^ * i 


- ' ''••• '. ‘ ' •* ■ 
.s;t V!^i 



* ! j'. % I 


>V:r^ni'n 

■iw . - ■- •* - ■••■" • 


^‘%w- 


^ L. 


M>V ^ <”• ■>: ’'i O’ * 





jj5t-iiiir0ti'-r;.>r :''s^?s -aiLPM 

i-iyifej', 'iff-; :' .;> 


2 !!Kii& 







■.I* .■». 



n;i. itTi .V 


I t 



' y 

''“W,i^r,ji<M:':(^s, 




ir ami .••■'. 

.7 > ' 

Ji . \ ^ _ 


I.U-. r.i,'.».;ili'?rfc,i»->fk-.*<‘ jW 

►Mil 









.D 1 


tiJN^i.;'^« 4ii<,v, i-'i..- .'-.■ V 

'‘t'l^-^*'**’*'^W* :*.- ‘ J\- ‘ii.M 'i.* 

'-'Wm 'Ufif^- .,,v. ,f|.<f.^^ 



^ ■' — 


tiV--;*;:/- ■•■; ■■ 


_ 1 ?) 


r# 



I»■ - I ■ ., <» 





' ■#. ■ - 










Chapter 4. Case Study 3—Dynamic Outcomes 
Management®* 

Introduction 

The primary purpose of DOM—^renamed LifeWings'^^ subsequent to our study—is to 
increase patient safety, reduce medical errors, and improve the quality of health care.^^ DOM 
achieves this by improving trainees’ skills in team-building, recognizing adverse situations, 

• 'X'7 

counteracting the effects of stress and fatigue, communicating, and decision-making. DOM 
provides interdisciplinary team training to surgeons, nurses, and anesthesiologists. The program 
draws heavily on Crew Resource Management (CRM) training from aviation , and was 
developed by Crew Training International (CTI), which offers specialized training programs for 
aviation, construction, general business practice, and the medical industry. 

Case Study Approach 

For DOM, we were able to access several sources of information. Specifically, we reviewed 
all instructional materials for the course that was observed. Relevant research studies that have 
been published on the effectiveness of DOM training were also reviewed. In addition to our 
document review, we observed two administrations of the course at a regional university medical 
center and collected pretraining data on student experiences and expectations and posttraining 
data on student reactions to DOM. Each of these activities is briefly described in more detail 
below. 

Document and Literature Review 

Our document review identified one article that has been published on the DOM approach. 
This paper was published by Rivers and colleagues and describes an evaluation of the 
effectiveness of this training. Results of this paper are described when we review the additional 
evidence that supports DOM. 

In addition to reviewing the literature, we carefully reviewed the Student Guides from the 
regional university medical center course. CTI staff provided these guides to us when we 
attended and observed the training. 

Course Observations 

AIR staff observed the two sessions of DOM training that were administered to hospital staff 
at a regional university medical center in September. Attendees were largely from trauma, 
emergency room, and cardio-thoracic medical areas. Over three-fourths of the attendees were 
nurses in addition to a few physicians, scrub technicians, paramedics, and receptionists. The 
same two CTI staff members taught the course each day. Instructors were retired Navy pilots 
who had also flown commercial aircraft. Observations were made of the classroom portion of 
this training, which lasted 8 hours each day. 


* Note: Appendixes cited in this report are provided electronically at 
http://www.ahra.QOv/qual/teamtrain/index.html#app . 


37 





Pre- and Posttraining Assessment Tools 

In addition to observing the classes, AIR staff administered pre- and posttraining assessments 
to all DOM participants. The pretraining tool was designed to assess student experiences and 
gauge participant attitudes and beliefs about teamwork. The results from this measure were 
described in an earlier section of this report entitled, “Description of Participants.” The 
posttraining tool measured participant utility reactions to DOM training. 

Instructor Interviews 

Unlike MedTeams and MTM, DOM instructors were not interviewed. This was primarily 
a function of the fact that DOM does not use a train-the-trainer technique. All DOM instructors 
are CTI employees. Therefore, many of our interview questions regarding instructor selection 
and training were irrelevant. However, although no formal interviews were conducted, we did 
have informal discussions with representatives from CTI about various aspects of the course. 

Below, we present the results from our analysis of this information. These results are 

'T'\ A 

organized around the same set of variables that we used to assess MedTeams and MTM. For 
each variable we provide descriptive data from our literature review and then findings from our 
observations, and data collection, when appropriate. Table 4 (p. 18-19) provides a summary of 
this information for DOM. 


Results 


Theoretical Basis 

The core objectives, curriculum, and instructional strategies for DOM were derived from 
CRM training programs that CTI has developed for the U.S. Air Force, U.S. Navy, as well as a 
number of other commercial airlines and military clients. The CTI developers believe that there 
are a number of aviation safety practices that can be applied directly to health care. Prior to 
conducting the course, DOM staff spends several days at the hospital site observing and 
conducting informal interviews with staff. Information gained from these activities is used to 
customize the DOM training materials to the particular institution in which the training will be 
administered. 

Analysis of Training Needs 

We were unable to identify any documentation that described how DOM was developed. 
Other than its basis in CRM and the fact that DOM developers spend time at the institution in 
which the training will be implemented, we could find no evidence of any other needs analysis 
activities (e.g., closed case reviews, critical incident study, etc.). We suspect that much of this 
training was derived from existing course materials that CTI uses to conduct CRM training for 
the military and commercial airlines. A review of these materials demonstrates its strong basis in 
aviation. However, we have no evidence to confirm or disconfirm this hypothesis. 

Training Objectives 

The primary objective of DOM is to teach aviation safety practices to health care workers. 
The DOM developers believe that there are many parallels between these industries that make 
CRM training relevant. These include stress, the need for highly functioning teams, the 
importance of accurate and precise communications, and the high cost associated with systems 
failures.^^ 


38 


Training Content 

DOM consists of seven modules that are taught using an 8-hour lecture, discussion, and small 
group activity format. They are: 

1. Introduction 

2. Managing Fatigue 

3. Creating a Team 

4. Recognizing Adverse Situations 

5. Cross-Check and Communication 

6. Decisionmaking 

7. Performance Feedback 

The “Introduction” module is a relatively short module that introduces the participants to 
DOM training, CRM, and the idea of aviation safety practices and their application to health 
care. The core skills that are the target of DOM training are also introduced. These include team 
management, recognizing adverse events, communication, decisionmaking, and performance 
feedback. Module 2 covers “Managing Fatigue.” Here, participants learn to recognize how 
fatigue can affect their performance and specific countermeasures for managing the negative 
effects of fatigue. Module 3 covers the topic of “Creating a Team.” Here, the benefits of 
teamwork are presented and discussed. In addition, participants engage in a small group activity 
in which they compile lists of things that team leaders and team members should and should not 
do. Module 4 focuses on “Recognizing Adverse Situations.” Here, the notion of red flags is 
described, and participants learn what they should do when they observe a red flag (i.e., see it, 
say it, fix it). Several case studies are then reviewed as a group. Each case study involves 
identifying red flags and solutions to these potential problems. Module 5 focuses on “Cross¬ 
checking and Communication.” Here, the steps in the communication process are presented as 
well as the four parts of an assertive statement. Again, participants are given an opportunity to 
apply the information presented during a case study exercise at the end of the module. Module 6 
covers “Decisionmaking.” Here, the different types of team decisions are described, as is the 
concept of building a shared mental model. Participants then apply these concepts during an in- 
class role-playing exercise. Finally, Module 7 focuses on “Performance Feedback.” Here, 
specific strategies are provided regarding how to debrief an individual’s or team’s performance 
as well as request feedback jfrom a teammate. 

Instructor Selection, Training, and Preparation 

As mentioned previously, unlike MedTeams and MTM, which employ a train-the-trainer 
format, CTI staff conducts DOM training. The two instructors who conducted the course were 
both retired Navy fighter pilots and also had worked for commercial air carriers. We did not ask 
these individuals how they were trained or how much preparation was involved, but we observed 
high reliability in how the training was conducted. In our opinion, CTI instructors were 
extremely professional, well trained, and highly engaging. 


39 


Instructional Strategies 

Description. DOM purports to employ a variety of training methods that address two of the 
three recommended phases of CRM training: Awareness and Practice-and-Feedback. The 
Continual Reinforcement phase is not addressed. 

The awareness phase includes 8 hours of classroom instruction. Once the classroom-training 
portion of the course is complete, CTI staff work with hospital personnel to develop checklists 
and other tools that can be used to promote safety at the hospital (the Practice-and-Feedback 
phase). One of the CTI trainers reviewed the basic materials with us and showed us examples 
that had been developed for other clients. However, we never received any copies of this 
information for a more in-depth review. 

Below, we describe our findings from reviewing the classroom component of DOM— 
Awareness Training. Although we would have liked to observe the actual applications of the 
posttraining tools and checklists, this was not possible because of time constraints. 

Findings. We observed two complete classes. Both classes started at 8 o’clock in the 
morning and concluded by 4 o’clock in the afternoon. Classes were kicked off by the Associate 
Dean for Clinical Affairs, who emphasized the university health center’s goal of becoming the 
Nation’s safest hospital. The same two CTI instructors then conducted the training. Instructors 
were consistent in their presentation of the materials across the two sessions. Effective use was 
made of the videotapes demonstrations, in-class exercises, and cases studies, which were 
designed to support the lecture portion of the training. 

Tnv >f 

Similar to MedTeams and MTM, we asked students to indicate the extent to which they 
agreed with the statement, “The training was well organized.” Overwhelmingly, students agreed 
or strongly agreed with this statement (see Table 7). The vast majority of students also reported 
that the training content was appropriate for their department. 

Training Effectiveness 

Description. For the classroom-based phase, DOM relies upon the collection of trainee 
reactions (Kirkpatrick’s Level I data) and a Human Factors Attitude Survey to determine training 
effectiveness. Regarding the reaction measure, participants were asked to rate how useful each 
DOM training module was. In addition, participants were asked to assess the quality of the 
instruction provided by each instructor. Regarding the Human Factors Attitude Survey, 
participants were asked to complete this survey twice, once prior to training and once after 
training. This measure appears to be modeled on the Cockpit Management Attitude questionnaire 
developed by Helmreich and colleagues,^ ^ though we received no direct information on the 
measure’s development, its subscales, or its psychometric qualities. However, CTI did not 
withhold this information from us; we merely did not request it because we viewed a 
psychometric analysis of this tool as beyond the scope of the evaluation. 

Findings. Similar to our investigation of MedTeams™ and MTM, we collected additional, 
independent data on trainee reactions to the course. Utility reaction data^^ were collected from 
participants. In addition, we reviewed existing empirical literature that has been reported 
regarding Kirkpatrick’s Level II (Knowledge), Level III (Behavior), and Level IV (Results). We 
reviewed an evaluation of the implementation of DOM that showed that the trainees developed 
positive attitudes toward the importance of teamwork, rated the training as useful, and 
demonstrated a 50 percent reduction in surgical count errors. Below, we describe our findings 
from these activities. 


40 


Level I Data, Attendees of the DOM courses had positive reactions to the course. Not only 
did they like the content and understand it clearly (98 percent agreement), they also felt that the 
training course was useful. Almost all (96 percent) felt that they could use the knowledge they 
learned on the job and perform the tasks learned (95 percent). Because of the training, they also 
felt more confident about their ability to work effectively in a team (92 percent) (see Table 7). 
Finally, the course met most attendees’ expectations (98 percent), mainly because of the 
informative content and the skills (e.g., working effectively as a team and communication) that 
were taught. 

When asked if they would make any changes to the course, the majority of attendees would 
not change anything about the course. Other attendees would like to include more area-specific 
scenarios and practical applications, such as role-plays, and more information on assertiveness 
advice and effective communication skills at different levels (e.g., RN, MD, etc.). A few 
attendees would like to see a more diverse group of students. For example, they would like to see 
technicians, nurses, nurse practitioners, and physicians from different areas mixed better at the 
different tables. In addition, a few attendees would like to have a re-evaluation done in the future 
to learn if the principles taught have been integrated and to see changes in best practices at other 
institution following the course. 

Finally, consistent with their positive impressions of the course, almost all attendees (98 
percent) would recommend the course to coworkers. The main reason that they would 
recommend the course focused on the team building content of the course, including the 
communication portion, which can help make individuals and teams be more effective in their 
jobs and improve patient care. In addition, participants highlighted the ability of DOM to change 
peoples’ mindsets. They also pointed out that participants should be able to integrate the skills 
that were learned into their jobs. 

Level II, III, and IV data. Data concerning the development and evaluation of DOM beyond 
trainee reactions are limited. As of January 2003, over 160 surgical staff members at Methodist 
University Hospital in Memphis, TN, had completed DOM training. An evaluation of DOM at 
Methodist Hospital found improvements in participants’ attitudes toward the importance of 
teamwork issues in the OR, favorable reactions concerning the usefulness of DOM training, and 
a 50 percent reduction in the number of surgical count errors. However, the small sample size 
makes it difficult to assess the generalizability of the results. Moreover, the lack of control 
groups makes it difficult to determine whether the training caused these improved outcomes. 


41 


Table 7. Posttraining opinions about Dynamic Outcomes Management^. 



Total (n=78) 

Item 

Mean 

Std. 

Dev. 

Agreement 

Neutral 

Disagreement 

1 am confident that 1 understood 
the training content 

4.6 

.65 

98% 

1% 

1% 

The training was well organized. 

4.7 

.72 

97% 

f 

3% 

1 am confident that 1 can use the 
knowledge that 1 learned on the 
job. 

4.6 

.68 

96% 

3% 

1% 

1 am confident that 1 can perform 
the tasks that were trained. 

4.4 

.70 

95% 

4% 

1% 

As a result of this training, 1 feel 
more confident about my ability to 
work effectively In a team. 

4.5 

.73 

92% 

7% 

1% 

The training content was 
appropriate for my department. 

4.4 

.83 

91% 

6% 

3% 

Training prepared me to work 
effectively in my job. 

4.4 

.78 

88% 

11% 

1% 

Training was an effective use of 
my time. 

4.4 

.86 

88% 

9% 

3% 

Training will help my department 
Improve patient safety. 

4.4 

.83 

87% 

10% 

3% 


Strengths and Weaknesses 

Strengths. Our review of the literature, observations of the classroom phase, and post 
training assessment suggest that participants had positive reactions to this training. Participants 
indicated that the training was well organized, and they felt that they could use many of the 
strategies discussed during training upon returning to their jobs. Second, DOM instructors were 
extremely professional and conducted high-quality training. Although not necessarily the most 
practical approach to implementing training throughout a large-scale organization like the DoD, 
using professional instructors from the course vendor resulted in significantly better and 
consistent instruction. Third, DOM staff relayed to us that there was no delay (like 
MedTeams™) between the classroom phase of DOM and when the safety tools are implemented. 
Therefore, skill decay is less likely with this program. Finally, the DOM developers are 
beginning to make efforts to collect additional data on DOM effectiveness beyond trainee 
reactions. Pre- and posttraining attitude data are currently being collected and discussions with 
the developer indicated that future studies are planned to examine DOM effectiveness. The 


42 



















results from the Methodist Hospital Investigation, which showed a reduction in sponge count 
errors, are encouraging. 

Weaknesses. Nevertheless, DOM, like the other programs, does have its limitations. First, 
we could uncover no evidence that the results of an in-depth pretraining needs analysis drove the 
development of DOM. It is our impression that the course developer extracted this information 
from CRM training and subject matter experts customized the materials to health care. CTI staff 
then visit the hospital in which training will be implemented to make any additional 
modifications to the courseware that is required. Second, like MedTeams and MTM, a primary 
objective of DOM training is to develop team skills. However, most of the classroom instruction 
focused on mastering declarative and some procedural knowledge. There wasn’t much time 
devoted to skills practice. Third, DOM did not employ a cultural assessment/evaluation 
component prior to implementing the training. As a result, it is entirely possible that DOM is 
effective only in hospitals that have already made a commitment to patient safety. The kickoff by 
the Associate Dean emphasized the medical center’s commitment to DOM and patient safety. 
Also, the pretraining data suggested that the organization has a culture that supports teamwork. 
Finally, although we did not specifically collect this data, the costs of implementing DOM are 
likely to be higher that MedTeams™ and MTM. This is primarily a function of the fact that CTI 
relies on its own cadre of instructors to conduct training and a full array of consultative services. 
While this produces reliable, high-quality instructors, we question the viability of such a strategy 
when training must be delivered in a timely fashion to multiple hospitals. 

Summary 

In conclusion, the DOM course was extremely well received. There was great support for 
DOM training and a strong commitment to patient safety by that organization. The content 
covered important aspects of teamwork and presented similar strategies to those discussed during 
MedTeams™ and MTM training. In our opinion, the quality of instruction was perhaps the best 
of the three programs we reviewed; however, there are many practical limitations with using 
vendor instructors when introducing medical team training in the DoD. Finally, we could not 
uncover much information about how the DOM course was developed, which caused us some 
concern. There was no evidence to suggest that DOM was in fact targeting the right skills for 
development or that the training objectives were appropriate. 


43 



.V! »;■ -^^5 ,r 


r- •! 






V 


■'•n 


«r. 

« 


j>i-. 


ra*' 


• ' » , • I 



'HiHr' * r'* 


I 1 




♦ s 




i'; 








tTl ' ^ ' > a’ll^ »V& ' '’b 

^ l.‘t ^ t, .• ’''^’Ui. • .iL^i . . ' -.i. ' ».'! “' 1- L' .’fl iV ■ « 






VV 


• g 





'rti,-,,v.,:‘^i-' ■.. W :V'^;r:ftfe'*«!■■;•:' 4viii'''’> ‘ ti '■5vrfc'~-J 

i:>. 1 . ^.■<k4.’^' .iji. ' I > at . T.‘ . J . t ^ ■D’.'^ .2 U'ttlff' 





■ 4 . 


jjj^,,.„, -?-8rU:nt‘<. l ' “, ' kiiii 

‘.'r-wiafVj T 

■M''''r'’‘: '** *' ,^y' 


^r'.;. ., ' ■. .t5j 


1 I. 





7lNf-^.>* r+*»il 







.«\'n 

:'^*^.:f 'I vx-tN't 

i,/ itt • <*• •' • tr-•"'■'I'^; '^' ’' 'n-* 

" 8 (i!, ,^««'-S;ii--j 


!1 


v.( 


■’itHP'^'%''’ #■:-'»»' '■) ’■-^i ‘ i' -liS ‘"’'«rr-«siti®d 



5l5};rf'/^?.'?';»‘4"-„ ^f ,V v y*A:.4';'in .^,:j' 


■#/ 5»n 




^ )ii4 ,-!'■■ •' CK,! .. t 


/.■W' i''>, 

•a-: ." Il 


d:^ \ut;i wfrt'<iJi^l4^!*- >fi 

. . . , ___ ^ " ■■■-'■..r '-.-.-j'i 


^■' '|,'<^'‘:.-'''^B$^i ■'.'/■'■:»r r'’'-r''^t'’:v •' ■’■ ■■.•• 

.. ■,.•.•••.■• . ■>. j 



. 


*!S '•y '■"' 


;.. ■■'■I-' 









t 


rl;v-■•-»T?» 

5lL4i?tfvfT---( ^ fi 

ur-JK 

^■'* ' 


i-V. 






41 ^ J 


i. Ie 








Chapter 5. Discussion* 

Introduction 

In this section, we summarize the results from our case study analysis of DoD-sponsored 
medical team training programs. As noted earlier, our primary goal throughout this project was 
to provide an independent and object assessment of MedTeams™, MTM, and DOM (now called 
LifeWings'^), based on the information we had access to. Because the vast majority of this 
information was qualitative in nature, we have purposely refrained from making comparative 
judgments among the three training programs. From time to time, we have pointed out strengths 
and weaknesses based on the information we collected. This was done to provide the DoD with 
as much information as possible. However, most of the programs’ differences were minor in 
nature and overall we felt that these programs were fairly equivalent for addressing the 
awareness phase of training. Therefore, the conclusions and recommendations that follow apply 
equally to MedTeams™, MTM, and DOM. 

Summary 

Are the Medical Treatment Facilities Ready for Training? 

The success of any training intervention depends not only on the training content and 
instructional techniques, but also on how the intervention is positioned, supported, and 

0 o 

reinforced by the organization. In particular, congruence between the objectives of training and 
the organization’s safety culture is critical to ensuring the transfer of trained behaviors. A safety 
culture is defined as a shared belief about the importance of safety, which leads to specific safety 
norms that dictate behavior within the group. Previous research by Helmreich and colleagues^^’"^^ 
has demonstrated the importance of a positive safety culture in both aviation and health care. 

Despite the fact that publicly-available instmments have been developed for assessing the 
safety culture of health care organizations, ' we were unable to locate any evidence that 
MedTeams™, MTM, or DOM conducted a pretraining safety culture assessment. As a result, it 
is entirely possible that CRM-derived team training programs only work in medical facilities that 
have already made a commitment to teamwork, secured upper-level management support, and 
recognized the need for change. Therefore, we recommend that future patient safety initiatives, 
whether it’s a training intervention or otherwise, always conduct a thorough pretraining safety 
culture analysis to identify barriers that can mitigate the transfer of trained skills. 

Is Aviation Crew Resource Management the Right 
Starting Point for Medical Team Training? 

Several researchers have recently suggested that health care providers look to CRM in 
aviation as a model for reducing medical errors. The argument for adopting CRM training is 


* Note: Appendixes cited in this report are provided electronically at 
httD://www.ahra.QOv/aual/teamtrain/index.html#app . 

^ The AHRQ Hospital Survey of Patient Safety Culture ( http://www.ahrq.gov/qual/hospculture/ ) was released in 
September 2004, after the work on this study was completed, and thus is not included in the references. 


45 





based on several important similarities between medicine and aviation. For example, both 
domains require multidisciplinary teams of highly-trained professionals to perform complex and 
dynamic tasks; both are conducted in high risk environments where the consequences of error 
can be deadly; and both require the team members to make decisions based on limited 
information under conditions of high stress, high workload, and time pressure. Moreover, recent 
research by Helmreich and colleagues'^^ has shown that pilots and health care professionals 
exhibit similar attitudes, such as the mistaken perception of invulnerability to the effects of stress 
and fatigue. Given these similarities, it is not surprising that all three programs are based on the 

Tnv >f 

aviation CRM model. Specifically, MedTeams is based on the U.S. Army’s CRM training 
course for helicopter crews; Medical Team Management (MTM) is based on the U.S. Air 
Force’s CRM training program for aviators; and Dynamic Outcomes Management 
(DOMyLifeWings'^'^ is based on military CRM training for fighter and cargo pilots. 

Despite these similarities, there are a number of important differences between aviation and 
medicine. For example, the size and composition of teams varies greatly between the two 
domains. Whereas commercial flight crews include 2-3 pilots whose performance is largely 
dictated by standard operating procedures, medical teams include up to 15 physicians, assistants, 
nurses, technicians, and staff from other disciplines whose performance is less amenable to 
standardization.'^^ In addition, because of their repetitive and highly scheduled nature, flight 
operations are well-suited to certain CRM tools such as prebriefings/debriefmgs, checklists, and 
quick reference cards.'^^ By way of comparison, certain medical operations—such as emergency 
medicine—typically cannot be planned in advance. This may make them ill-suited to 
interventions which require such strict, scheduled, and orderly procedures. 

In addition to these differences, it is important to note that CRM is not a universal remedy. 
CRM by itself will not eliminate all the systematic contributors to medical error. Rather CRM is 
one component of a comprehensive approach to improving patient safety. For example, in 
AHRQ’s Evidence-based Practice Report (Number 43), Making Health Care Safer: A Critical 
Analysis of Patient Safety Practice, 50 safe patient practices or areas for system improvements 
were identified, one of which was CRM. Furthermore, this report noted that the evidence base 
for CRM’s effectiveness was viewed as low relative to others, but the report also suggests further 
research on the introduction of CRM in health care is likely to be beneficial. 

The differences between aviation and health, as well as the limited evidence base, do not 
negate the potential value of introducing CRM training in medicine. Rather, they highlight the 
importance of carefully tailoring CRM principles and practices from aviation to medicine and 
the need for future empirical investigations that demonstrate CRM’s effectiveness. This tailoring 
should be driven by the results of a thorough needs analysis. 

Have the Needs Analyses Gone Deep Enough? 

All training programs should begin with a comprehensive needs analysis. At a minimum, this 
should include an organizational analysis (to identify the organization’s readiness for change), a 
personnel analysis (to identify the workforce’s specific knowledge, skill, or attitude 
deficiencies), and a task analysis (to develop and sequence the learning objectives).'^^’^^ 

Presently, at least two of the three major team training programs—MTM and 

A 

MedTeams —^began with at least a partial needs analysis. For example, MTM was developed 
as a result of ineffective teamwork and communication as a recurrent theme in major medical 
events. A subsequent validation of the need occurred with a review of 60 critical incidents, 
which revealed communication problems were the single largest contributing cause of medical 


46 




T*'X A 

errors. Similarly, MedTeams began by a systematic review of closed case records, which was 
scored using a teamwork failure checklist to identify trends.^ However, subsequent versions of 
MedTeams™ have involved customizing the original ED curriculum to L&D and the OR by 
adapting the case studies and examples to the clinical context with the aid of subject matter 
specialists. MTM has not engaged in such customization and has advocated a one-size-flts-all 
approach. DOM has employed a similar approach to MedTeams™. 

We question whether or not the MedTeams™ and DOM strategy will adequately address the 
unique team requirements of different medical services. Our expert panel suggested that 
variation is likely so we recommend additional, deeper-level analyses when customizing medical 
team training to a specific specialty area. Alternatively, we see some merit in the MTM 
approach, if a generic set of skills exists that cuts across all medical specialties. However, these 
skills and the behaviors they represent would have to be established and agreed upon. 

Are the Current Instructional Strategies Appropriate? 

Despite improvements in training technology, our case study analysis found that all three 
programs rely almost exclusively on classroom-based instructional techniques—such as didactic 
lectures, case studies, videotaped vignettes, and pretraining readings—to deliver their course- 

nr\yf 

specific content. For example, MedTeams includes an 8-hour block of classroom instruction 
that contains an introduction module, five learning modules, and an integration unit. 
Supplementary materials include a 30-minute video that depicts examples of good and bad 
performance. After completing the classroom training, each team member participates in a 4- 
hour practicum that involves practicing teamwork behaviors and receiving feedback from a 
trained instructor. Coaching, mentoring, and review sessions are also provided during regular 
work shifts.^ Similar instructional strategies are currently used by MTM and DOM. Our review 
of the published literature suggests that these strategies are best suited to developing factual or 
declarative knowledge of CRM principles,^ ^ such as that which occurs during awareness training. 

To date, none of the programs use simulators, such as the type used in Anesthesia Crisis 
Resource Management ^ to provide trainees with the opportunity for skills practice with 
instructor feedback, as recommended in aviation CRM. Although both classroom-based and 
simulation-based strategies are justifiable vehicles for delivering the training content, the sense 
that we gained from our review was that classroom-based techniques are used primarily because 
they have always been used in the past, not because they are particularly well-suited to specific 
teamwork-related competencies. Therefore, we recommend that future research in MTT address 
which instructional strategies are most appropriate for which type of competencies, and then 
tailor the training such that the instructional strategies are chosen to specifically complement the 
training content. 

Are the Current Methods for Delivering Ciassroom Training 
Sufficient? 

Each program has two primary instmctional components: a classroom-based (i.e., 
implementation) component and post-classroom (i.e., sustainment) component. In this section, 
we address the classroom-based component. Our document review revealed that both 
MedTeams™ and MTM use a “train-the-trainer” paradigm, whereby individuals from the 
medical treatment facilities receive instruction in teamwork skills, and then return to their 
respective organizations to train the rest of their colleagues. However, our review also suggested 
that MedTeams™ and MTM have few formal mechanisms for ensuring consistency and 


47 



preventing performance degradation among instructors once they return to their respective 
facilities. To some extent, this issue is not relevant for DOM, because a small cadre of trainers 
from Crew Training International (CTI) conducts all the training. 

TTVI 

For example, upon successfully completing the train-the-trainer course, MedTeams 
instructors are then “certified” by Dynamics Research Corporation. Unfortunately, there is little 
publicly available information about this certification process. For example, it is unclear whether 
the certification is based on the instructors’ knowledge of teamwork principles, their actual level 
of teamwork skills (e.g., communication, decision-making, etc.), their ability to teach teamwork- 
related issues to others, or some combination of these factors. MTM has even less stringent 
procedures for ensuring consistency and preventing performance degradation. It is requested (but 
not required) that potential MTM instructors have at least 5 years of clinical experience in their 
specialty areas, at least 1 year of time remaining in the Air Force, that they be competent 
speakers, and that they have previous experience delivering training. 

The presence of formal procedures for ensuring consistency during the implementation phase 
is essential to the success of any training program. However, our observations and interviews 
suggest that the implementation of MTT programs may not always correspond to their 

T^lVyf 

developers’ specifications. For example, our observation of the MedTeams course showed that 
the classroom portion of training varied from instructor to instructor, with some courses lasting 
only 3 hours. We discovered similar results while interviewing MTM instructors. 

To some extent, this is to be expected with the train-the-trainer paradigm. Specifically, the 
train-the-trainer paradigm is designed to be a cost-effective means for distributing training to a 
large number of people within a short period of time. However, the trade-off is that the training 
developers have less control over how the training is actually implemented at the host sites. 

DOM has taken an alternate approach. By using a small cadre of trainers from Crew Training 
International, they can reach a much smaller audience; however, this provides them with greater 
control over the quality of instruction. Because our observations and interviews were based on a 
small, nonrepresentative sample, it is impossible to estimate the actual base rate of instructor 
deviations from the prescribed training syllabi. Nevertheless, we caution MTT developers to 
carefully and periodically monitor the implementation of their respective MTT programs to 
ensure that the training is delivered as designed. 

Are the Current Methods of Sustainment Sufficient? 

In this section, we address the post-classroom component of training. Our document review 
revealed that all three programs have some form of post-classroom follow-up. During this time, 
trainees are provided with the opportunity to practice their trained skills, and to receive some 
degree of feedback from their site coordinator. 

For example, the post-classroom component of MedTeams lasts for approximately 6 
months. During this time, trainees are encouraged to practice their newly-trained skills. 

According to our document review, MedTeams™ employs a number of tools for sustaining 
effective team performance, such as trainees’ monitoring one another’s performance. In addition, 
routine team meetings are conducted to ensure team members’ continued focus; status boards are 
used to maintain team members’ situational awareness about particular patients, nurses routinely 
participate in morbidity and mortality (M&M) meetings, and performance evaluations directly 
consider teamwork issues.^^ MTM includes similar techniques for sustaining and reinforcing 
teamwork-related issues in the post-classroom environment, such as periodic (scripted) safety 
drills, periodic team leader meetings, formal recognition of improved/effective teamwork during 


48 


the trainees’ annual performance reviews, and a report on implementation progress to the Air 
Force Patient Safety office.^^ DOM also includes such practice with the customized development 
of checklists and other tools that support safety practices. 

Previous research suggests that the presence of formal procedures for sustaining trained 
behaviors in the posttraining environment is essential for ensuring the success of training.^ ^ 
However, our review suggests that the respective course developers spent considerably more 
time focused on the short, classroom-based component of training (which typically last between 
8-12 hours) than on the extensive, post-classroom component (which can last up to 6 months). 
This is ironic, given that the classroom phase was focused largely on the awareness of teamwork- 
related issues. For each program, the classroom component covered teamwork-related 
knowledge and attitudes; there was substantially less opportunity for actual skills practice with 
instructor feedback. Therefore, we urge MTT developers to pay greater attention to the post¬ 
classroom component of training, because this is where the real changes in learning (i.e., 
development of teamwork-related skills), performance (i.e., greater use of checklists, pre- 
briefings, cross-checking, etc.) and results (i.e., reductions in medical errors) are likely to occur. 

Recommendations 

In this section, we provide theory- and practice-based recommendations for improving 
medical team training. The overarching theme in this section is the need for a more thorough 
understanding of the medical team performance and medical team training domains than 
currently exists. 

standardize the Knowledge, Skills, and Attitudes 

A recent review by Cannon-Bowers and colleagues^" has noted that the team skills literature 
is confusing, contradictory, and plagued with inconsistent labels and definitions. In some cases, 
different labels are often used to describe the same teamwork competencies; in other cases, the 
same labels are used to describe different competencies. Thus, our first recommendation is to 
develop a standard nomenclature that names and defines the core teamwork-related knowledge, 
skills, and attitude competencies for successful teamwork in health care. 

We envision this first taxonomy as all-inclusive, in the sense that it would incorporate the 
core competencies that potentially influence all types of medical teams, regardless of specialty or 
context. In other words, the proposed taxonomy would be medical-team generic. This medical- 
team generic specification connotes two theory-driven steps: determining an appropriate level of 
explanation for identifying core teamwork competencies and determining which of these 
competencies are relevant to medical teams. 

The first step, determining an appropriate level of explanation, is necessary to ensure that the 
constructs included in the taxonomy are conceptualized broadly enough to span the entire health 
care field, yet specific enough to facilitate valid measurement. Further, although this list of core 
competencies should reflect all relevant aspects of team performance, it must be concise enough 
to generate teamwork and team training research and to facilitate team training needs analyses in 
organizations. 

The second step, determining relevant core competencies, encompasses two activities. One is 
to establish which of the many competencies manifested in previous research are relevant to 
virtually all medical teams; a second, and perhaps more demanding task is to identify core 
medical-team competencies that have not emerged from team research in other domains. For 


49 


guidance in this area, researchers might rely, to some extent, upon medical experts like those 
engaged in the panel that AIR held in January 2003 (refer to Appendix D for a list of panel 
participants). 

However, we believe that, along with developing a theory of medical team performance, 
standard job-analytic techniques, such as task questionnaires, structured interviews, and non- 
obtrusive observations will yield the most valid information. Moreover, we emphasize the 
importance of large-scale, stratified data collections because the goal is to identify generic 
competency requirements with which the medical community at large concurs. 

Identify Practice-Specific Training Requirements 

We believe that the medical-team generic taxonomy described above would be useful to both 
theoretical and applied research; nevertheless, we believe that no single set of team knowledge 
and skills can be applied equally across all medical practices and contexts. For purposes of this 
discussion, we define a “practice” as a medical specialty or subspecialty, such as emergency 
medicine, general or family medicine, intensive care, surgical medicine, obstetrics, and so forth. 
Medical practices differ dramatically across a variety of criteria, including team size, lifespan, 
acuteness (i.e., consequence of error), and member composition, to name but a few. As an 
example, emergency medicine providers function in hospital emergency departments, in 
emergency-response mobile units, and on battlefields. Similarly, urban and rural general 
practitioners operate in independent or multipractitioner offices, as well as in community walk-in 
clinics. Neither the competencies that impel successful teamwork nor an optimal team training 
strategy can be expected to generalize across these diverse contexts. And, of course, not all 
members within the same team will necessarily need the same knowledge, skills, and attitudes. 

Thus, our second recommendation is to identify practice-specific training requirements for 
diverse health care contexts. These taxonomies would not be redundant with the generic 
taxonomy. Rather, a practice-specific taxonomy would reflect “core” competencies in the sense 
of denoting the knowledge, skill, and attitude requirements that are central to teamwork in a 
given practice. The medical content and procedures that define that practice would drive the 
identification of relevant team-competency requirements. 

Virtually no previous research has addressed the manner in which differences within and 
between medical practices should be reflected in practice-specific taxonomies. Yet we find this 
issue sufficiently compelling to warrant further investigation. Because these taxonomies are 
derived from the medical characteristics of specific practices (and the contexts within them), 
subject-matter experts who represent each practice might be invaluable in identifying practice- 
specific team competencies that are not redundant with the generic taxonomy. Nevertheless, we 
would also suggest that researchers avail themselves of job analysis questionnaires, structured 
interviews, and nonobtrusive observations. 

Leverage Existing Knowledge About Teamwork Training 

Although the field of medical team training is still in its infancy, the science and practice of 
team training is far from new. Research on the design, implementation, and evaluation of team 
training programs began nearly 50 years ago. Since that time, a substantial body of research has 
been conducted regarding the most effective strategies and techniques for training teamwork- 
related knowledge, skills, and attitudes. For example, Salas and his colleagues have compiled an 
extensive collection of principles and guidelines for assertiveness training,^^ cross-training,^^ 
stress management training,^^ and team self-correction.^^ 


50 


Unfortunately, the existing medical team training programs do not appear to have leveraged 
this body of research. For example, as noted earlier, the MTT programs that we reviewed rely 
almost exclusively on classroom-based training methods, rather than choosing from a variety of 
instructional strategies to complement the specific training content. With few exceptions, new 
advances in training technology—such as computer-based training, low-fidelity simulations, 
embedded training, and scenario-based training—have rarely been used, despite growing 
evidence regarding their effectiveness.^^ Recent advances in training theory—such as the effect 
of pre- and posttraining factors on training outcomes, the effect of practice schedules on skill 
acquisition and retention, and the critical role of individual differences in shaping trainees’ 
motivation—have similarly been ignored. Each of these factors has been shown to 

improve the effectiveness of team training programs. Thus, our third recommendation is that 
instructional designers leverage such information in their own medical team training programs. 

Develop a Standardized Training Specification 

__ 

Two different MTT programs are currently sponsored within the DoD: MedTeams and 
MTM (DOM is not funded by the DoD but has been implemented at one naval medical hospital). 
As we have shown in previous sections, these programs use similar instructional strategies, have 
similar training content, and have demonstrated similar results concerning their effectiveness. 
These similarities beg the question: Why have three separate MTT programs? Despite outward 
appearances, this is not a trivial question. Practically speaking, multiple training programs could 
be problematic, especially during deployment when intact teams are split up. For example, a 
deployed team may consist of some individuals who have received MTT and some who have 
not. Alternatively, a deployed team may include staff from multiple branches of the armed 
forces, each of which may use a different MTT program. This could create a great deal of 
confusion, for example if the team members were practicing similar teamwork behaviors, but 
calling them by different names or using different approaches to accomplish the same task. 

Standardizing MTT would produce greater consistency in the design, delivery, and 
evaluation. Thus, our fourth recommendation is that the DoD develops a training specification 
for MTT programs in specific practice areas. In addition to describing the core teamwork 
competencies for each practice area, the specification would outline the appropriate instructional 
strategies for each core competency, the appropriate sequencing of training activities, and 
outcome measures for assessing the degree of skill acquisition, to name but a few. Moreover, 
such a training specification would allow the DoD substantial flexibility in procuring and 
administering the training. For example, training could be administered internally or with the aid 
of contractors, depending on time and resource constraints. Regardless of who offers the training, 
the content and delivery would be essentially standardized by the program specification. 

Develop Technical Assistance on Crew Resource 
Management Issues in Health Care 

The Federal Aviation Administration (FAA) promotes aviation safety by two primary 
mechanisms: education and enforcement. Of the two, education is the most proactive way to 
prevent problems from occurring. One of the FAA’s primary means of educating the aviation 
community on important initiatives is through the use of advisory circulars (ACs). In general, ACs 
present guidelines for developing, implementing, and evaluating various FAA initiatives such as 
pilot Crew Resource Management (CRM) and simulation-based training programs. 


51 



Advisory circulars are designed to be informative. They typically present one or more 
ways—^but not the only way—of addressing a particular issue. For example, the FAA’s advisory 
circular on CRM training includes a definition of CRM concepts, fundamentals of CRJM training 
implementation, suggested curriculum topics, guidelines for assessing the effectiveness of CRM 
training, example behavioral markers for specific CRM skills, and suggestions for further 
reading. 

Under the Patient Safety and Quality Improvement Act of 2005 (P.L. 109-41), the Agency 
for Healthcare Research and Quality (AHRQ) has been charged by the Secretary of HHS with— 
among other things—^providing technical assistance to the newly created Patient Safety 
Organizations (PSOs) that will support providers in the “improvement of patient safety and 
reduce the incidence of events that adversely effect patient safety.” We recommend that the 
Agency prepare technical assistance documents comparable to the FAA Advisory Circulars to 
help the PSOs guide providers on issues related to team training and error prevention. We 
believe that human factors-related technical assistance to the PSOs would go a long way to 
educate the medical community about the importance of MTT for ensuring patient safety, and for 
ensuring consistency across MTT programs. 

Conclusions 

In conclusion this report presents an in-depth case study analysis of three medical team 

’T'K/T 

training programs, MedTeams , Medical Team Management, and Dynamic Outcomes 
Management®. This was the first independent assessment of these programs. The case study 
approach allowed us to collect detailed, comprehensive information on each program, which we 
reported along a common set of variables. Although this study was qualitative in nature, it is the 
first effort to capture that state-of-the art in medical team training. 

AHRQ’s Evidence Report 43 (Making Health Care Safer: A Critical Analysis of Patient 
Safety Practices)^^ suggested that future research on medical team training is likely to be 
beneficial and have a significant impact on patient safety. We view the results presented in this 
investigation as a starting point for future studies on medical team training. Here, we have 
provided information on current objectives, strategies and successes of existing programs as well 
as where opportunities for improvements exist. We have also delineated several areas where 
future research is most warranted. However, empirically-based research will require a mandate 
from program sponsors, Federal agencies, or the health services research community; greater 
access to health care workers and patients to collect both process and outcome data; and 
significant resources in terms of time, money, and personnel. Nonetheless, we believe that such 
investments are worthwhile, because few would dispute the relation between team performance 
and safety. The challenge is to show irrefutable evidence that substantiates the relation between 
teamwork in health care and the desired outcome, a reduction in errors, because the medical error 
rate, although unacceptably high, has a relatively low base rate. We believe that this can be 
accomplished under the right conditions and point to the on-going L&D study as an example of 
the kind of investigations that are required. 

In summary, we believe that the future is bright for medical team training because there is an 
existing knowledge base from aviation and other high-risk industries on which health care can 
gain traction. Great strides have been made with the introduction of the development and 
introduction of the three training programs described in this report. We view these programs as 
the first generation of what we believe will be continuous, sustained advancements in medical 
team training over the next decade and beyond. 


52 





References 


1. Kohn LT, Corrigan JM, Donaldson 
MS, editors. To err is human: 
building a safer health system. A 
report of the Committee on Quality 
of Health Care in America, Institute 
of Medicine. Washington, DC: 
National Academy Press; 2000. 

2. Doing what counts for patient 
safety: Federal actions to reduce 
medical errors and their impact. 
[Report of the Quality Interagency 
Task Force (QuIC) to the 
President.] Rockville, MD: Quality 
Interagency Task Force; February 
2000. 

3. Helmreich RL, Foushee HC. Why 
crew resource management? 
Empirical and theoretical bases of 
human factors training in aviation. 
In: Weiner EL, Kanki BG, 
Helmreich RL, editors. Cockpit 
resource management. San Diego: 
Academic Press; 1993. p. 3-45. 

4. Gregorich SE, Wilhelm JA. Crew 
resource management training 
assessment. In: Weiner EL, Kanki 
BG, Helmreich RL, editors. 

Cockpit resource management. San 
Diego: Academic Press; 1993. p. 
173-98. 

5. Federal Aviation Administration. 
Crew resource management 
training. Advisory Circular 120- 
51 A. Washington, DC: Federal 
Aviation Administration; Feb. 10, 
1993. 

6. Gaba DM, Howard SK, Fish KJ, et 
al. Simulation-based training in 
anesthesia crisis resource 
management (ACRM): a decade of 
experience. Simul Gaming 
2001;32:175-93. 

7. Simon R, Morey JC, Rice MM, et 
al. Reducing errors in emergency 
medicine through team 
performance: the MedTeams 
project. In: Scheffler AL, Zipperer 


L, editors. Enhancing patient safety 
and reducing errors in health care. 
Chicago: National Patient Safety 
Foundation; 1998. p. 142-6. 

8. Stone FP. Medical team 
management: improving patient 
safety through human factors 
training. Military Health System 
Health Care Reengineering. HCR 
Reference No. 00080; 2000. 

9. Goldstein I. Training in 
organizations: needs assessment, 
development, and evaluation. 3rd 
ed. Pacific Grove, CA: Books/Cole; 
1993. 

10. Grove DA, Ostroff C. Program 
evaluation. In: Wexley K, Hinrichs 
J, editors. Developing human 
resources. Washington, DC: BNA 
Books; 1990. 

11. Kirkpatrick DL. Evaluation of 
training. In: Craig RL, editor. 
Training and development 
handbook: a guide to human 
resource development. New York: 
McGraw-Hill; 1976. p. 18.1-18.27. 

12. Kraiger K, Ford JK, Salas E. 
Application of cognitive, skill- 
based and affective theories of 
learning to new methods of training 
evaluation. J Appl Psychol 
1993;78(2):311-28. 

13. Alliger GM, Tannenbaum SI, 
Bennett W, et al. A meta-analysis 
of the relations among training 
criteria. Personnel Psychol 
1997;50(2):341-58. 

14. Alliger GM, Katzman S. 
Reconsidering training evaluation: 
heterogeneity of variance as a 
training effect. In: Ford JK, 
Associates, editors. Improving 
training effectiveness in work 
organizations. Mahwah, NJ: 
Erlbaum; 1997. p. 223-46. 


53 


15. Kraiger K, Jung K. Application of 
cognitive, skill-based and affective 
theories of learning to new methods 
of training evaluation. In: Quinones 
MA, Ehrenstein A, editors. 

Training for a rapidly changing 
workplace: applications of 
psychological research. 
Washington, DC: American 
Psychological Association; 1997. p. 
151-76. 

16. Baldwin T, Magjuka RJ. 
Organizational context and training 
effectiveness. In: Ford JK, 
Kozlowski SWJ, et al., editors. 
Improving training effectiveness in 
work organziations. Mahwah, NJ: 
Lawrence Erlbaum Associates; 
1997. p. 99-128. 

17. Donabedian A. Explorations in 
quality assessment and monitoring. 
The definition of quality and 
approaches to its assessment. Arm 
Arbor, MI: Health Administration 
Press; 1980. 

18. Coyle YM, Battles JB. Using 
antecedents of medical care to 
develop valid quality of care 
measures. Int J Qual Health Care 
1999;11(1):5-12. 

19. Morey JC, Simon R, Jay GD, et al. 
Error reduction and performance 
improvement in the emergency 
department through formal 
teamwork training: evaluation 
results of the MedTeams project. 
Health Serv Res 2002;37(6).T 553- 
81. 

20. Rivers RM, Swain D, Nixon WR. 
Using aviation safety measures to 
enhance patient outcomes. AORN J 
2003;77(1): 158-62. 

21. Gregorich SE, Helmreich RL, 
Wilhelm JA. The structure of 
cockpit management attitudes. J 
Appl Psychol 1990; 75:682-90. 

22. Smith-Jentsch KA, Jentsch F, 

Payne S, et al. Can pretraining 
experiences explain individual 


differences in learning? J Appl 
Psychol 199;81(1): 110-6. 

23. Simon R, Langford V, Locke A, et 
al. A successful transfer of lessons 
learned in aviation psychology and 
flight safety to health care: the 
MedTeams system. Patient Safety 
Initiative 2000: spotlighting 
strategies, sharing solutions. 2000 
Oct 4; Chicago, IL: National 
Patient Safety Foundation; 2000. p. 
45-9. 

24. Simon R, Salisbury M, Wagner G. 
MedTeams: teamwork advances 
emergency department 
effectiveness and reduces medical 
errors. Ambul Outreach, Spring 
2000;21-4. 

25. Simon R, Morey JC, Locke A, et 
al. Full scale development of the 
Emergency Team Coordination 
course and evaluation measures. 
Andover, MA: Dynamics Research 
Corporation; 1997. 

26. Morey JC, Simon R, Jay GD, et al. 
A transition from aviation crew 
resource management to hospital 
emergency departments: the 
MedTeams story. In: Jensen RS, 
editor. Proceedings of the 12th 
International Symposium on 
Aviation Psychology; 2003 Apr 14; 
Dayton, OH, Dayton, OH: Wright 
State University Press; 2003: p.1-7. 

27. Dynamics Research Corporation. 
MedTeams: labor and delivery 
measures guide. Andover, MA: 
Dynamic Research Corporation; 
2002. 

28. Dynamics Research Corporation. 
Evaluation of a teamwork 
intervention in labor and delivery 
units. Andover, MA: Dynamic 
Research Corporation; 2002. 

29. Searles RB. Patient safety program 
educating medical community. 
AirforceLink Online News. 2002. 
http://www.af mil/news/Jul2002/n2 


54 


0020709_1082.shtml. [No longer 
accessible as of 1/26/2006.] 

30. Kohsin BY, Landrum-Tsu C, 
Merchant PG. Medical team 
management: patient safety 
overview. Unpublished training 
slides; 2002. 

31. Kohsin BY, Landmm-Tsu C, 
Merchant PG. Medical team 
management: increasing patient 
safety through human factors 
training. Military Psychology . 
2002. Bolling Air Force Base, DC, 
Air Force Medical Operations 
Agency. 

32. Kohsin BY, Landrum-Tsu C, 
Merchant PG. Medical Team 
Management agenda, homework, 
observation/debriefing tool, and 
lesson plan. Bolling Air Force 
Base, DC: Air Force Medical 
Operations Agency; Unpublished 
training materials. 2002. 

33. Kohsin BY, Landmm-Tsu C, 
Merchant PG. Implementation 
guidance for Medical Team 
Management in the MTF [Medical 
Treatment Facility]. Bolling Air 
Force Base, DC: Air Force Medical 
Operations Agency; Unpublished 
manuscript; 2002. 

34. Kohsin BY, Landmm-Tsu C, 
Merchant PG. Medical Team 
Management: commander's call 
briefing. Bolling Air Force Base, 
DC: Air Force Medical Operations 
Agency; Unpublished training 
materials. 2002. 

35. Kohsin BY, Landmm-Tsu C, 
Merchant PG. Medical Team 
Management: MTF [medical 
treatment facility] implementation. 
Bolling Air Force Base, DC: 96th 
medical group; Unpublished 
training materials. 2002. 

36. Crew Training International. 
Lifewings"^^. Available from 


http://www.cti- 

crm.com/products/medical.php 

37. Lifewings"^^. Saving lives and 
reducing costs with proven aviation 
tools. Available from 
http://www.saferpatients.com/index. 
htm. 

38. Salas E, Cannon-Bowers JA. The 
science of training: a decade of 
progress. Ann Rev Psychol 
2000;52:471-99. 

39. Helmreich RL, Merritt AC. Culture 
at work in aviation and medicine: 
national, organizational, and 
professional influences. Brookfield, 
VT: Ashgate; 1998. 

40. Sexton JB, Thomas EJ, Helmreich 
RL. Error, stress, and teamwork in 
aviation and medicine: cross- 
sectional surveys. Br Med J 
2000;320:745-9. 

41. Gaba DM, Howard SK, Fish KJ, et 
al. Simulation-based training in 
anesthesia crisis resource 
management (ACRM): a decade of 
experience. Simul Gaming 

2001 ;32(2): 175-93. 

42. Leape LL. Error in medicine. J 
Amer Med Assoc 
1994;272(23):1851-67. 

43. Pizzi L, Goldfarb NI, Nash DB. 
Crew resource management and its 
applications in medicine. In: 
Shojana KG, Duncan BW, 
McDonald KM, et al., editors. 
Making health care safer: a critical 
analysis of patient safety practices. 
Evidence Report/Technology 
Assessment No. 43. AHRQ 
Publication No. 01-E058. 
Rockville, MD: Agency for 
Healthcare Research and Quality; 
2001. p. 501-10. 

44. Schaefer HG, Helmreich RL. The 
importance of human factors in the 
operating room. Anesthesiology 
1994; 80(2):479. 


55 


45. Leedom DK, Simon R. Improving 
team coordination: a case for 
behavior-based training. Mil 
Psychol 1995;7(2):109-22. 

46. Thomas EJ, Helmreich RL. Will 
airline safety models work in 
medicine? In: Rosenthal MM, 
Sutcliffe KM, editors. Medical 
error: What do we know? What do 
we do? San Francisco: Jossey-Bass; 
2000. p. 217-34. 

47. Wiener EL, Kanki BG, Helmreich 
RL. Cockpit resource management. 
San Diego: Academic Press; 1993. 

48 Shojana KG, Duncan BW, 
McDonald KM, et al., editors. 
Making health care safer: a critical 
analysis of patient safety practices. 
Evidence Report/Technology 
Assessment No. 43. AHRQ 
Publication No. 01-EOS8, 

Rockville, MD: Agency for 
Healthcare Research and Quality; 
July 2001. 

49. Goldstein IL, Ford JK, editors. 
Training in organizations. 4th ed. 
Belmont, CA: Wadsworth, 2001. 

50. Wexley KN, Latham GP. 
Developing and training human 
resources in organizations. New 
York: HarperCollins; 1991. 

51. Salas E, Rhodenizer L, Bowers CA. 
The design and delivery of crew 
resource management training: 
exploiting available resources. 

Hum Factors 2000; 42(3):490-511. 

52. Federal Aviation Administration. 
Advisory circular 120-54: 
advanced qualification program. 
Washington, DC: Federal Aviation 
Administration; 1991. 

53. Anonymous. Formal teamwork 
training improves teamwork and 
reduces emergency department 
errors: results from the MedTeams 
project. 1999. Lansing, ML 


54. Cannon-Bowers JA, Tannenbaum 
SI, Salas E, et al. Defining 
competencies and establishing team 
training requirements. In: Guzzo 
RA, Salas E, editors. Team 
effectiveness and decision-making 
in organizations. San Francisco: 
Jossey-Bass, 1995; p. 333-80. 

55. Smith-Jentsch KA, Salas E, Baker 
DP. Training team performance- 
related assertiveness. Person 
Psychol 1996;49:909-36. 

56. Volpe CE, Cannon-Bowers JA, 
Salas E, et al. The impact of cross 
training on team functioning: an 
empirical investigation. Hum 
Factors 1996;38(1):87-100. 

57. Driskell JE, Johnston JH. Stress 
exposure training. In: Cannon- 
Bowers JA, Salas E, editors. 

Making decisions under stress: 
implications for individual and 
team training. Washington, DC: 
American Psychological 
Association; 1998. p. 191-217. 

58. Smith-Jentsch KA, Johnston JH, 
Payne S. Measuring team-related 
expertise in complex environments. 
In: Cannon-Bowers JA, Salas E, 
editors. Making decisions under 
stress: implications for individual 
and team training. Washington, 

DC: American Psychological 
Association; 1998. 61-87. 

59. Salas E, Dickinson TL, Converse 
SA, TS. Toward an understanding 
of team performance and training. 
In: Swezey RW, Salas E, editors. 
Teams: their training and 
performance. Norwood, NJ: Ablex; 
1992; p. 3-29. 

60. Salas E, Caimon-Bowers JA, 
Blickensderfer EL. Team 
performance and training research: 
emerging principles. J Wash Acad 
Sci 1993;83(2):81-106. 

61. Salas E, Bowers CA, Cannon- 
Bowers JA. Military team research: 


56 


10 years of progress. Mil Psychol 
1995;7(2):55-75. 

62. Federal Aviation Administration. 
Line operational simulations: line 
oriented flight training, special 
purpose operational training, line 
oriented evaluation. Advisory 
Circular 120-35B. 9-6-1990. 
Washington, DC: Federal Aviation 
Administration; 1990. 






:4i(n' .« ’VMT,; 

!.■ ? 
4 -* * 


•T’ ,-.-V 




‘A. 


r-i-, 


iii • ' 

ZL* '* *■ “n' 

r ^ ■' v"; .'i ■. 


;|]f^.^">t'r'-; ^) »,,v **:it 

. • -!.f 

* ' U,' .-“ . ^ ;■ j , • ,y‘ 

'j .. __ 

-r* u 

‘4 ' 

‘,‘-,r .' 


\^yv • ' “a-V 

,.-*' ■ i». 




■!>-,l, .'t 

<1 


af»/ 


■«*T' 


^ 1^'- 


^ 'C' 

, ' -i T*** ^ 


'‘^i,v!.'\’■^™flkiT'!''' <■ ivV 1 !• .'^i|t. <’.i6n* 

/ \. . fi‘ ’ . Jrt* • . 'I ;. , 

< •. * » .v '^ •>',:• .K 


A 





«!*'• i^‘ * \ ^ 


fri'. 


•?,*. .V. Tv\»"vr -^ 


..j’ -- 


l' ^ ' 


'\ i' ‘ 





rayt' Mtos;,:, 

K" -> ■'■' V**A;',' , '' ' ':■ 


V:, V .* 

Lj^r* '.t •A.'*' 









A 


^r.-,^ '«.^' f. >»_*■C.r'*'." 

1 '*•< *■ ■ j' t " * ' - 


*■' i 




- S' 


-V 


rj 

.j^ .kA k. 




»t ^ ^ 1 







vlnw 

•« . 4 






i?:,'ivf, 'fcflR 

* ' ''.iClfi? JiEiV 


■< . ..fVA‘; rVpV-lltejjillflfci* 



» .' 






IMF ^ .'4 V r..' 4 * 

‘ ' j */' t*r -* 'i ■' 




1 /. 



k .I*- 


Vf/* t • 



*>♦? f 





■f *..' 


f.-'' .w_ 



P I ‘ *,• iS, 

;\^,,.:t^. 4=^ 

_., >4f%. ' 

f /y* jfc••**' * •♦Ji 4 ^''' 



]4euk4 v%--4' 

V\..aaUilr5. 


I 



^ • k f’ 


.R 



■ V'i:j'’. 


. 4 ■•■;■ » 


^ 


:‘V';M_I ■ ' ;,'*-r" ^ 


»' rV 


c,:., 



'iivi t’ ! 


.* * 











U.S. Department of Health and Human Services 

Mike O. Leavitt, Secretary 

Office of Public Health and Science 

Richard H. Carmona, M.D., M.P.H., F.A.C.S., Surgeon General of the United States 

Agency for Healthcare Research and Quality 

Carolyn M. Clancy, M.D., Director 




LC ACQUISITIONS 















































4^ ^ JI> vO' LI 'O' *^^^0 it x^ 

f 


Jk IW^M ':; s*”^. 

« Xja *0 » i-* kC> 

\\ o® 







i> 



<t >^ 

Oo ■'■ 




'C'N' - i ''j>‘i>' “ 

•. <?6 
•r 

_ ^ A,' - < 

<r 

° <r s» ’y. « 


•J 

" C?'" \\7^‘>‘^X 

% % C? * °0 -i^ .<• 






.4 


. “* o.:\’’'^ • ‘ r ;%"°'‘° ^j?-^, 

■^. /^V/h, "" - ^ . 


0^, s V ' " * 

: .xv^i 



<^\ 

^ f * '^oV' <■ ■* ^ 

“ V >P^,i*, t 

❖ O 

. r*.^ /N'' 






y'^ J^y ^ * 

I 



>n 


o 

y^A "^0^1 ]k.'^ -t* ^ " .4^ 

V "'o> 

.= \^m.: ‘ 


° 'focfi u 

o 





ffn 


% 

\ i 

\ 

a ts. ^ ^ 4 y 

"o 

x* jA h 1> '-t^ 

■>bv^ * ^ 

>P< 


<<r 
u 

r 4"% 

*y ^ ■^t’ ^ x» ^ tf> ® .a^Qa « 

vV ^ f' - 



ffn »>^ 



ty h 



.v> -5- 

-$■ £,ong, 't'o 

. cP 4 »^ 5 SS^ ' °0 

Oq ^ 



° I 

■^0-7 * ^ 


4>V*0,'% 

/»o\/ 

" xV'^ 


-^^x vw^/^-r ^<? 

‘ o^ ^'N 

, cr Pi /!■ 

P-O^ <• 

* " 

O, 'VX, ’ ” ■ Q^° S 0<,f %. *’“N°'’ 

z “ 

° o » x^JV., 

ax^ Ty*^ I// sa ur ^ -oP '*^5K, 

# ' O - 4 ^ <- * "^' * ♦ o * >■ o N o 4 

■'.o v4 cP.* - 

'^ov^ ^> 0 ^ ° 

» . 0 ^. <p « 




V< 


O^ 



<A J^' 


» %. * 

•‘kcT '^. "’A 0 - ^ o 

d^ co«®<^»* *-( v?* 

«■ AP A-. O * 

■ • <-:v ■' ’ ’.o*-. • • • ^>t. ' 


A aO-’ ^■*’»o^o*<vV Pd. *• 








■••“’ • 0 .V*"’ •>?>•• ^.V*"■°^>■*■!. * 0 .%* 






»* > .^wr/ -v^^\ % 

ko'^ ° ^ "^o"^ 

AQ - 




« 

» . »* i>^ 

s * 

’.r 

^V \ ° 

Xp Pt> **’'^ « * '^ v^ 

qV c°'^°*f 4?^ ^ 

'»bv^ : 

• 4 » O ^K •A'rs 

% »°>iO'^\<^ . 0, '• 

»’Pj. " '^- -v^A'‘A‘’*.'Hi. 



: : 

° 4 '% t 

.■" xT 0„ * 

» 0 K 00 ^ 

• , j(<\W/)Jt, <0 %, 



z.Ptk^*'''-* * 


V / 


z 'j-4= - 

o <V A> o 

sr K ^ 

* '^. <' 7 v 7 s' A 

d^ oO""*. "td 4 ?^ ■ 

y «> ,-t<^ ..^d> '-f'i a"^ ❖ 


'A* o. -dC? ^ 

V ’P'^J* " 

V* 






O AO * 


4 




^ _A> 

6 ^ c 0 ^® 4 x 


a\ * L I * P'a ^ 



aVA ^ 



^ Vj 





















O 



^ .... < ‘ 



-/»1? »* 

IJ. *''»‘°\^'^^t.0,\'' 

. c, 


.- ~ 


■''3 

; * 


c^ '^i 

k.,^ 



'' %. ^ 

%. '^j-4’ “ 

\ %HP» \ 

cON<=«, V * ’ 'lA' "'*/V° ’ ' 

«> '^OV o 




Jffi'l «v «,<»*>. V-V ^ ® ^< ° ^ 'i « O ^ ^ 


/ /\ %W." ^ 

“ ‘.'V" * ’ ’>% * ^' */V° ‘"V 

%A A ^ .^XsSXvTSjk 4/* ^ A» m^l//6'^^ ^ *4jti ^ ^ ^ 

» r ‘ ’^ov" o - T '■ 

^o<- * 







.M»' 'O' 


<^' '•i^ O 

f« ^ «4fvS^^S«^ <r \j 

o 

<» 



AO. 


^^K!l^s3^ >*- ^ «^.^%Z4'^ A O '»’ ^ /-N ^ 

JTosoO 

5 fst.- ,.#.<1 ■ 


Vk ^ ^ O 4 

a ct^ ^ ^ ^ 

Sfc ^ apfOTTpL^ Cj j 



; * 
j* 4 !^'*^ * 

cF C°«°<<, \ ^ t. 

<y »* ,'^^«>.s» ' *A^ / 

° ft '»nA ft 

■Cl 


Vv. “'O 

ft - ■"t 

,, ft o AO^ 

* 


■ftx 
■* % 

<0 ^ 

<?>. ftOftV"'.''* .i 

,* 'v. « 


K 'ft’^s«^/ ^ * . 

-->;''-°’>^!-orv 

■^\ ' 

o^ cO^’^'if. '^o ^ ^ 

. ft cr °o .-*• 

a ^ ^ ftSi^^ovxtr^ i/> . <c^ 

>Cr 



4> X 


<t U- 





V\^P'ft',.ftft'‘’\’ft^^.--" -Jft •.«s^.- »ft- ■■% ivxK-' ft' 

■•.5V“ ■ '>!tr''X‘ ■ ■ ' • ft<‘i;^',S'’ ■ ■ oA. 

O AO^ ftO-A ft^^^° A*^ ^yO-<U "S^^O AoL 

° “#R* -"^^1 “"fife; %■# " 

°^W» t®i^“ A'V eS>J'p " 

rx o Wft« 8 ft\V/ a/ 4> ^ ^aiZ>i3PQ*? ■^ 4 . o iZ/KsWr **' » 

^ 4>v ^ ^^ ^ ^ ^ 

A ^ ^ ^ 

y ° »> 4> "r>^ O 

•y 




'ZVftA''^f'>fto'‘'1'M'r\vft<--^^^ ■ ■^' '* 

l/> 

J* 


%^i 

X 

.-* c?^’ * 

O x"^ ^ _ 

■'■ft *^00 

</> 




J 

Heckman 

'i 

" i 

j. 

' 

J 

BINDERY, INC. 
Bound-To-Please* 


07-B2021 

1 


N. MANCHESTER, INDIANA 46962 



‘V" V^oft> 

* 


r. ft ° -A°^ t 

cy e,^ ^ -0^ * O^ 

- 




•e 


o 

l»- 

ft-^*^ ftO 




■$.°, ft ft ^ V a\, ft o,P| 



o ^ y* 


A -f'p - 

<>\ 


A, 

•. * 

■r '■^*'-eP .ft 

^4. ^<!r ^ ^ ^ "^y 

'^ Tvv /V^ ^ 

-j: • -.V • • A-^X ■ Vf‘ 

c^ .0 


O A- 

•Cl 



^ 'A '■ 

v-<y 




















































